Internal
(done) Code review MRR PR #812 β completed 4 review rounds on customer identity resolution refactor (DAT-471). Decomposition of itg_mappings__customer_identities into 5 sub-models. Identified critical issues: Zoho-only rows silently dropped by WHERE clause, NULL domain NOT IN semantics, Zendesk fan-out risk. PR merged Mar 10. (PR #812)
(done) Continue commenting DE-201 (PR Workflow enhancement) β completed review and documentation of CI/CD gaps in internal-aml-project repo. Proposed tiered roadmap: quick wins (PR templates, CODEOWNERS), expanded test coverage, new automation (lineage diff, impact analysis). (Notion)
dim_contacts β HubSpot Contact and fct_customer_stats β HubSpot Company. Validated MAR stays within free tier limits (~1,215 records/month vs 3,500 limit). Follow-up noted for next week: check Census usage estimate to confirm free tier eligibility.viz_settings (PR merged Mar 17).n_records column missing, PR #841). DAT-555 still open: fct_job_queue_performance model is stale and likely to be removed; mart_product__dataset_datamodel_dimensions also under review. Several skipped downstream models identified: itg_customer_support__organizations, mart_customer_support__organizations, mart_product__activities, and others.isolatedStorage: true), R2 stream consumption, and schema docs. (Release)shadcn/ui skills β Integrated shadcn/ui skill from the official repo into the agent-skills system. Synced remote skill files and created symlink. (PR #13)1on1-report.sh automation for generating retrospective reports from Logseq notes.fct_job_queue_performance and write Notion doc for team review.DONE Investigate failed dbt assets in #data-ops-bot (DAT-569)
mart_event__drill_down and mart_event__view_underlying_data).dbt Mart Snowplow Events job ran before dbt Mart Product (which builds dim_users), causing null tenant/user IDs. Secondary bug: LEFT JOIN to internal_tenants incorrectly passed internal events when user was missing from dim_users.coalesce fallback. Wrote Notion investigation doc (DE-207) with debugging guide for juniors.INFORMATION_SCHEMA (excluding dev_* in prod).DONE Review Thuan’s work on DE-208 / mart_product__dataset_datamodel_dimensions (PR #845 β OPEN)
bridge_product_retailer) from Redshift pipeline and Holistics AML layer_migration_baseline, _bridge_backup) created on Redshift.retailer_id + retailer_current_pricing_skey columns to atc_price_history, backfilled ~27M rows (75.7% coverage). Known gap: 8,713 keys from Chemist Discount Center slug mismatch β documented.sp_refresh_daily_prices, sp_refresh_atc_price_history_enrich, sp_run_all, sp_refresh_price_history_from_s3, sp_validate). Ran sp_run_all() β enriched table validated at 12.1M rows, join path works without bridge.feat/eliminate-bridge-aml β removed bridge from all 4 datasets + market_price_rank model, added direct rcp_skey relationships, deleted agg_retailer_product_price model. Pushed to Holistics, dashboards verified._bridge_product_retailer_backup (frozen, 222K rows) for reference. Remaining data quality items (near-miss slugs, missing retailers, history-only) handed off to Anurag.master_product β retailer_current_pricing β bridge β atc_price_history to master_product β retailer_current_pricing β atc_price_history (one fewer hop), improving query performance.airbyte_schema_*). Found critical price_history_skey collision: 2,774 keys collide across countries (39,710 rows), mainly Watsons across SG/ID/MY.data_source_name in AML routing per-hotel queries via JWT user attributes. Limitation identified: cross-database aggregation (50-60 DBs) requires a data mart. Moved to Apr 6.mart_product__dataset_datamodel_dimensions) β MERGED. Fix for PARSE_JSON syntax error. All CI checks passed after review rounds in W13.prefect.yaml, dbt model dependencies via ref() calls, and Airbyte sync frequencies.mart_event depends on mart_product (specifically mart_product__users and mart_product__dashboard_widgets), but runs weekly on Monday 00:15 before the daily product mart at 01:10. Safe because mart_event doesn’t use the + prefix (reads existing BigQuery data, no upstream rebuild).US-1099511640417 wasn’t mapped to Zendesk org in analytics. Traced through itg_mappings__zendesk_tenant β HubSpot domain mapping β Zendesk organizations table.source-zendesk-support:0.2.6) has a pagination bug capping ingestion at 10,000 records. The raw organizations table had exactly 10,000 rows; 680 of 911 org IDs in tickets were missing.domain_hubspot_companies, and PR #64 (Thuan) deleted that model. Git didn’t flag it because they touched different files.holistics.h-theme-builder.pages.dev/theme-builder β will try with more prospects.pages/Projects.md.people + land pattern: Iceland, Greenland, England, Switzerland, Finland, Kazakhstan, Uzbekistan (stan = land).Progress Review (Start/Stop/Keep):
DONE Fix failed dbt tests β snowplow events (DAT-569)
DONE Check Zendesk β Tenant Inovonics (DAT-571)
fct_zendesk_tickets β itg_mappings__zendesk_tenant β stg_zendesk__organizations β _airbyte_raw_organizations and found org 13430326317849 missing entirely.stg_zendesk__*.sql models). Waiting for review.WAITING DAT-555 β Fix fct_job_queue_performance BigQuery resource limit (DAT-565)
dim_exchange_rates table (DAT-576), implement dual metrics (constant_currency_mrr + reported_mrr), implement canonical customer_skey for long-term identity resolution.Sync Zoho Events failed Apr 12 with HTTP 403 (Zoho error code 104003) β OAuth token lacked ZohoSubscriptions.events.READ scope after Finance (Vincent, Sriram) adjusted admin permissions for Janice’s account.fct_job_queue_performance BigQuery resource limit (DAT-555)SELECT * in CTEs, fix tracking_utm_content data type.dom_calendly__events) + 1 mart (fct_calendly_events). Source migrated from Google Sheets to Fivetran. Bug fix: fct_sales_leads was dropping 88% of call-booking leads due to NULL UUIDs in Google Sheets β now joining on invitee_email_domain from Fivetran source. Impacts 10+ downstream dashboards (Leads Analytics, Sales Performance, Deal Activity, etc.).fct_sales_leads join.fct_mrr, handle FX noise in historical growth metrics (pin to end-of-month rate), validate against Zoho totals. Notion doc: MRR discrepancy ~30,000 SGD between Zoho (340K SGD) and bi.h (243K USD Γ 1.28 = 311K SGD).theme: 'light_theme' | 'dark_theme' to the embed payload. Also, Sankey chart labels don’t convert to light colors in dark theme (color palette not tied to theme). Basata says this is urgent β already rolled out embed to 3 clients, wants dark mode by end of next week.one_letter_codes table is a multi-purpose lookup where same code means different things per category (e.g., ‘C’ = ‘Cancelled’ for StayStatus, ‘City’ for Ledger). Proposed 3 approaches; decided on Approach 1 (SQL Query Models with inline joins) β most reliable, avoids bloating diagram view._dbt_max_partition gaps), dbt Cloud Canvas, and data ingestion tool comparison.AGENTS.md. Already practicing this with Logseq automation._dbt_max_partition gaps. Directly applied to DAT-555 fix.fct_sales_leads join.dim_exchange_rates with live rate ingestion. Manager priority #1.AGENTS.md, building automation scripts that learn from mistakes. This is a genuine competitive advantage in the AI-first era. Continue investing in it.dim_exchange_rates and constant-currency MRR will shortcut your DAT-576 implementation.AGENTS.md for different subdirectories.journals/2026_03_19.md covering Source / Staging / Domain / Mart / Deprecated / Fixed layers.mart_growth__calendly_bookings (view β fct_calendly_bookings, grain = event Γ attendee Γ rep) and mart_growth__calendly_events (table β dim_calendly_events).enabled: false): mart_growth__demo_calls, mart_growth__onboarding_calls, mart_core_business__demo_call_bookings, dom_calendly__demo_call_bookings, dom_calendly__onboarding_call_bookings.mart_core_business__sales_leads now joins dom_calendly__bookings directly on attendee_email_domain (resolves the 88% drop in call-booking leads found last week).unique_map_identities_zendesk_hubspot_zendesk_organization_id. Zendesk org 900032617243 (1win.pro) maps to 2 HubSpot companies because 1win.pro appears in all_domain_names of both HubSpot 5855457913 and 18891852445.IN UNNEST(all_domain_names) join in itg_mappings__zendesk_tenant.sql (and similar integration-layer joins) creates fan-out whenever a domain appears in multiple HubSpot companies. Originated from DE-187 phase 1 (PR #812).customer_skey work). Not yet actioned due to sick leave..sql file. Phase 3: drop the table..sql only stops refresh, leaving the table with stale data and silently misleading consumers. Either keep the wrapper alive or drop the table entirely.journals/2026_05_25.md (will scope after the holiday). Slack 1, Slack 2.dim_exchange_rates with live rate ingestion. Carry-over, manager P1.journals/2026_04_27.md.Two-holiday short week (Thu Apr 30 Reunification Day + Fri May 1 Labour Day). Effective workdays = Mon/Tue/Wed for Holistics work; Sat/Sun spent on Smartclass releases and reading. Net: shipped Calendly BI rollout + tenant-lookup fix on the Holistics side, and two Smartclass releases (v0.4, v0.4.5) on personal side. Next week’s MRR exchange-rate work is still zero-progress (3-week carry-over).
First full 5-workday week after W18 holidays. Big internal closure: Lead Funnel by Sales Motion (DAT-560) shipped β DAT-561/562/563/564 all closed May 8, ending a 5-week carry-over that had been blocked on Calendly. Big presales loss: [[Showbie]] went to Omni; learning synthesis captured. MRR exchange rate (DAT-576) still NOT STARTED β now 4-week carry-over and being treated as a restart in W20: Hieu left and most implementation context is gone, so W20 Mon AM is recovery + scoping (not implementation), with a Wed-EOD escalation gate to anh Triet. CircleCI dbt PR #858 still open (cherry-pick
dbt#861merged). DAT-555 closed for real this week. Tooling week was strong: Holistics MCP, amp plugin for MCP error logging, slackcli, caveman skill, RTK install.
Steady throughput week. Big wins: CircleCI PR review marathon finished (rounds 2β4, prefect#400 + dbt#858 closed), [[BuyCo]] custom theme rolled out on Volumes dashboard, Power BI playbook draft started for BuyCo onboarding (Tue May 19). Khai To’s 2-way sync CLI shipped β unblocks the local-dev demo video (4-week WAITING cleared). MRR exchange rate (DAT-576) still NOT STARTED β now 5-week carry-over: Mon AM recovery slot lost again to CircleCI close-out, week absorbed by BuyCo theme + Datacubed dashboard fix + Power BI prep. Honest call: needs hard escalation in next 1-on-1. Tooling: rg/grep + rtk fix, personal finance app rental-affordability simulation. Weekend = VCT Pacific watching + travel (no H1 self-reflection done β sliding W21).