Why Missing Data in Scheduled Exports Isn't a Minor Bug

When you build a scheduled data export, the assumption baked into most implementations is that if there’s no data for a time slot, you just skip it.

It’s a reasonable assumption. It’s also wrong.

Downstream systems — ERP platforms, global reporting tools, financial reconciliation engines — don’t treat a missing record and a zero-value record the same way. A missing record often means “we don’t know.” A zero-value record means “we know, and it was zero.” Those are fundamentally different states. Conflating them breaks downstream logic in ways that are hard to trace back to the export pipeline.

We hit this exact problem building the Datahub XML export system for New Zealand’s largest OutSystems platform — daily automated exports across cash sheets, quarter-hour brand sales data, and bank deposit reporting, all to a standardised global format, serving companies including McDonald’s NZ across 10,000+ daily users.

The fix was gap-filling logic. Every missing time slot gets an explicit zero, not a blank. Here’s why that matters and how we built it in OutSystems O11.

The quarter-hour sales data problem

The specific trigger was quarter-hour brand sales exports. Sales data comes in across the day in 15-minute intervals — 6:00am, 6:15am, 6:30am, and so on through close of business.

In a live restaurant environment, not every slot has sales. A quiet period mid-morning might have zero transactions. The restaurant management system records nothing for that slot.

When the export ran, those empty slots produced no records. The XML output jumped from 9:45am to 10:15am with nothing in between. The downstream global reporting system, built to expect a complete set of 15-minute intervals, interpreted the gap as missing data — not zero sales — and flagged the export as incomplete.

The business impact: exports were being rejected or requiring manual intervention to correct before they could be processed. On a daily scheduled export running across multiple restaurants, that’s not a one-off problem — it compounds.

Gap-filling: the concept

Gap-filling means generating placeholder records for every expected time slot, regardless of whether source data exists for that slot.

For quarter-hour sales data across a standard restaurant trading day, that’s a known, finite set of intervals. You can generate the full expected sequence programmatically — every 15-minute slot from open to close — and then join your actual sales data against it. Slots with sales data get the real values. Slots without sales data get explicit zeros.

The output is always a complete, continuous sequence. No gaps. No ambiguity for downstream systems.

How we built it in OutSystems O11

The implementation has three parts.

1. Define the interval structure

Start with a Structure in your Data tab — call it SalesInterval:

SalesInterval
├── SlotTime         (DateTime)
├── NetSales         (Decimal)
├── TransactionCount (Integer)
├── TenderBreakdown  (Text)  ← optional, depends on export spec

This is your template record. Every slot in the trading day will be represented as one of these, zeroed by default.

Then define a Local Variable in your server action as a List of SalesInterval — this becomes the sequence you build, populate, and eventually serialize.

2. Generate the expected interval sequence

In a Server Action, use a While loop to generate every expected slot:

Variables:
- CurrentSlot      (DateTime) = TradingDayStart
- IntervalMinutes  (Integer)  = 15
- IntervalList     (List of SalesInterval)

While CurrentSlot <= TradingDayEnd:
    NewSlot = Default SalesInterval
    NewSlot.SlotTime = CurrentSlot
    NewSlot.NetSales = 0
    NewSlot.TransactionCount = 0

    ListAppend(IntervalList, NewSlot)
    CurrentSlot = AddMinutes(CurrentSlot, IntervalMinutes)

AddMinutes is a built-in OutSystems function under the Date Time category — no extension needed. TradingDayStart and TradingDayEnd come in as input parameters to the action, driven by your tenant configuration.

The output of this loop is a complete zeroed sequence — every slot accounted for, nothing skipped.

3. Fetch actual sales data — once, before the loop

Before you do anything with the interval list, fetch all actual sales records for the period in a single Aggregate:

Aggregate: GetSalesForPeriod
- Source: SalesRecord
- Filter: SalesRecord.RestaurantId = RestaurantId
         AND SalesRecord.SlotTime >= TradingDayStart
         AND SalesRecord.SlotTime <= TradingDayEnd

Assign the result to a Local Variable: ActualSales as List of SalesRecord.

This is the critical performance decision. Do not run an Aggregate inside the While loop that populates your interval list. Every iteration would hit the database — on a full trading day at 15-minute intervals that’s 50+ queries per restaurant per export run. Fetch once, work in memory.

4. Populate the interval list from actual data

Now iterate through your IntervalList and match against ActualSales:

For Each Slot in IntervalList:
    MatchingSale = ListFilter(ActualSales,
        Value.SlotTime = Slot.SlotTime
        AND Value.TenderId IN AllowedTenderIds)

    If MatchingSale.Count > 0:
        Slot.NetSales = MatchingSale.Current.NetSales
        Slot.TransactionCount = MatchingSale.Current.TransactionCount
    // else: slot stays zeroed — gap-fill holds

ListFilter is your tool here. It returns a filtered list from an in-memory list without touching the database. If it finds a match, you overwrite the zeros with real values. If it finds nothing, the slot stays as initialised — zero values, slot present, gap filled.

One OutSystems gotcha: ListFilter returns a List, not a single record, even when you’re matching on a unique key. Check .Count > 0 before accessing .Current — accessing Current on an empty list throws a runtime exception, not a graceful null.

5. XML serialization — where the zeros can disappear

This is where most implementations quietly break.

OutSystems has no native XML serialization action that handles arbitrary structures cleanly. Your options:

Option A: JSON-to-XML via intermediary. Use the built-in RecordListToJSON to serialize your IntervalList to JSON, then transform to XML via an Extension or a custom Server Action. Straightforward but adds a transformation step and you need to control the output schema carefully.

Option B: Build the XML string directly. Iterate through IntervalList and build the XML string using string concatenation or a Text Builder pattern. More verbose, total control over output format, no surprises. For a fixed schema like a global data export spec, this is often the right call — the schema won’t change, so the verbosity is a one-time cost.

Option C: Forge component. The OutSystems Forge has XML serialization components — XML Records is the most commonly referenced. Evaluate against your specific output schema before committing; generic serializers sometimes have opinions about attribute ordering or namespace handling that conflict with strict downstream specs.

The zero vs null problem in serialization:

OutSystems will serialize a Decimal attribute with value 0 as <NetSales>0</NetSales> — explicit zero, present in output. It will serialize a null/NullDecimal as nothing, or as <NetSales xsi:nil="true"/> depending on your serialization approach.

Your gap-fill initialises every slot with 0 (not NullDecimal), so as long as you’re not reassigning NullDecimal anywhere in the population step, your zeros will serialize correctly. Audit your population logic to make sure — if MatchingSale.Current.NetSales can return NullDecimal from the database for a record that exists but has no value, handle it explicitly:

Slot.NetSales = If(IsNull(MatchingSale.Current.NetSales), 0, MatchingSale.Current.NetSales)

The tender type filter — Site Property vs Entity

We implemented the allowed tender type configuration as a tenant-level Entity row, not a Site Property.

Site Properties in OutSystems are global — one value per property across the entire environment. For a multi-tenant system where each restaurant has its own tender configuration, Site Properties don’t work. You’d need one Site Property per tender type per restaurant, which becomes unmanageable fast.

Instead: a TenderConfig Entity with columns RestaurantId, TenderTypeCode, IsIncluded. One row per tender type per restaurant. The export action fetches the allowed tender codes for the current restaurant at runtime:

Aggregate: GetAllowedTenders
- Source: TenderConfig
- Filter: TenderConfig.RestaurantId = RestaurantId
         AND TenderConfig.IsIncluded = True

Pass the resulting list of codes into your ListFilter in the population step as AllowedTenderIds. The filter and the gap-fill operate independently — a slot with zero sales in included tender types still gets an explicit zero. A slot with sales only in excluded tender types also gets a zero, because those transactions are correctly excluded from this export.

Keeping them separate matters. If you bake the tender filter into the gap-filling logic, you end up with slots that appear to have zero sales when they actually have sales in excluded tender types — a different and more confusing result than a genuinely quiet period. Clean separation, easier to test both in isolation.

Scheduling and the idempotency note

The export runs on an OutSystems Timer. Timer retry behaviour in O11 means if the action throws an unhandled exception, the platform will retry automatically — up to the retry limit configured in Service Center.

This means your export action needs to be safe to rerun for the same period. The gap-fill logic is inherently safe — regenerating a zeroed interval sequence is idempotent. The risk is in your file output: if you’re writing to a file drop location, a rerun will attempt to write the same filename again.

Handle this with a CreateOrUpdate pattern on your export log Entity (tracking which exports have been generated per restaurant per date), and either overwrite the output file deterministically or skip generation if a completed record exists. Don’t let the Timer retry silently produce duplicate export files in your drop location.

When you need this pattern

The signal is: scheduled export, time-series or interval-based data, downstream system with structural expectations.

Common cases beyond restaurant sales:

• Utility metering exports — hourly or half-hourly consumption intervals, expected to be complete even in zero-consumption periods
• Financial intraday reporting — trading hour breakdowns where gaps are ambiguous
• IoT or sensor data pipelines — device readings at fixed intervals where silence means “no reading” vs “reading of zero” is a meaningful distinction
• Payroll or shift reporting — period-based exports where missing periods need explicit representation

If your downstream system was built by someone else, to a spec you didn’t write, treat structural completeness as a hard requirement. The alternative is debugging rejection errors in a system you don’t control.

The broader principle

Scheduled exports feel like plumbing. Write a query, serialize to XML, drop a file somewhere. Done.

The problems show up at the edges — quiet trading periods, partial data, downstream systems with opinions about format. Gap-filling logic is one of those patterns that looks like over-engineering until the first time an export gets rejected for a missing 9:45am slot.

On New Zealand’s largest OutSystems platform, we built it in from the start. Daily exports run cleanly across every restaurant, every tender type, every quarter-hour slot — whether the restaurant was busy or empty.

That’s what production-grade looks like.