Notes
Good timing does matter
- Context: when I want to propose an idea to dbx team about setting up a Claude connector to interact with dbdiagram right inside claude desktop: Ampcode
- I can read it from each team’s sync to quickly decide whether to propose an idea
- For example, when I want to propose ((69c10475-619d-42df-b63e-1884b27ce548)), read the dbx team sync to find out whether they are having some similar key words in mind
dbt fails debugging process (collected from Thuan)
- Check log in prefect, view error log
- In model, there are 3 CTEs, run each cte step by step to find out where the error happen, check upstream
- Find out the root cause
- Proposing solutions
Surface interactive dbdiagram into claude/chatgpt (chat interface)
- Context, from these:
- https://claude.com/blog/interactive-tools-in-claude
- {{video https://www.youtube.com/watch?v=bluAmTHoEow}}
- That said, allowing an interactive interface directly inside claude chat, similar to slack or figma.
- Use case
- When doing data support tickets, which are mostly data modeling and AQL writing problems, I’d like a quick way to reproduce: by imagining the dataset diagram from customers’ questions (as natural language) or AML models/datasets
logseq.order-list-type:: number
- Input: questions (as text) or AML models/datasets (they copy-paste into the ticket) logseq.order-list-type:: number
- Output: an interactive interface of DBML diagram that lives inside the chat, can interact like logseq.order-list-type:: number
- Currently I am doing it with mermaid
logseq.order-list-type:: number
- However, mermaid can not drag-and-drop tables, can not rearrange tables, which is important to me because:
logseq.order-list-type:: number
- I want to focus on some tables (fact tables) to design metrics on them. logseq.order-list-type:: number
- Some diagram may contain hundreds of tables, which is hard to organize and focus on some tables logseq.order-list-type:: number
- Mermaid’s syntax of relationship is harder to read (in my opinion).
logseq.order-list-type:: number
- logseq.order-list-type:: number
erDiagram CUSTOMER ||--o{ ORDER : places ORDER ||--|{ LINE-ITEM : contains CUSTOMER { string name string email string address } ORDER { int orderNumber string deliveryAddress } LINE-ITEM { string productCode int quantity float pricePerUnit }
- logseq.order-list-type:: number
- An exporting to dbdiagram feature so that teammates can collab on logseq.order-list-type:: number
- However, mermaid can not drag-and-drop tables, can not rearrange tables, which is important to me because:
logseq.order-list-type:: number
- If I use DBML instead:
logseq.order-list-type:: number
- I have to manually copy-paste DBML code from claude into dbdiagram to visualize and vice versa, just to have the visual view logseq.order-list-type:: number
- I pop this idea out of my head because of this use case
- When doing data support tickets, which are mostly data modeling and AQL writing problems, I’d like a quick way to reproduce: by imagining the dataset diagram from customers’ questions (as natural language) or AML models/datasets
logseq.order-list-type:: number
- Benefits
- With an increasing number of users using AI, we can promote DBML more, pitch it as a powerful database designing and documenting language
- New acquisition channel
- Context, from these:
Tasks
Partly today
DONE Revise phase 2 of MRR problem
Discussed with anh Hieu
- After PR#812, these are statuses of problems:
- 1.1 and 1.2 (static exchange rate in dbt)
- 2.1 (missing holistics region when processing manual mapping logic, dbt): Fixed
- 2.2 (domain mapping between tenant & company)
- 2.3 (duplicated companies in hubspot - ICEYE case)
- 2.4
- 3.1
- 3.2
- 4
- Next step (according to anh Hieu) is:
- To align with growth team on the dashboard:
- Note: https://www.notion.so/holistics/Data-alignment-between-Data-Team-and-RevOps-326f89dc7e49800ebdd3e0f282391746
- Dashboard: https://bi.holistics.io/dashboards/v4/45480-customer-identity-monitoring?_t=tab_0ze3
- We push growth team (Quinn, Arden) to review this.
- In my opinion, the question still remains:
- Can I trust the numbers (MRR, customer count) on bi.h?
- If the numbers between bi.h and zoho are mismatched, what are the reasons?
- For those reasons, are they accepted by growth team?
- Action from me (Chinh): review what Hieu wrote, help him communicate with growth team.
- Fix currency exchange rate
- Action from me (Chinh): review what Hieu wrote in slack, help him communicate with growth team.
- To align with growth team on the dashboard:
- After PR#812, these are statuses of problems:
DONE Check around failed dbt tests (snowplow events) in #data-ops-bot
LEARNING The bigquery agent skill often get trouble in mapping logical (in dbt) and physical table names (in bigquery). How can I improve this?
- Take dynamic mapping table from
INFORMATION_SCHEMA(excludingdev_*if in prod)
- Take dynamic mapping table from
DONE Write notion document
DONE Implementation
- Waiting for anh Triet to take a look. Then tell anh Dong to review both PRs. When approved, add commits to event models to guard that.
- 2 PRs merged. Deploy, run backfill and run snowplow again.
- Done. Everything is running well.
- Snowplow run success: https://prefect3.holistics.dev/runs/flow-run/c1f2e4c8-c7df-48e7-b221-f3e0ab60c6bc
- Backfill success: https://prefect3.holistics.dev/runs/flow-run/558210a0-602b-44c6-9f22-57f2f8887885
Done
- DONE Teach Duc Anh
DONE Guide Thuan to solve the linear DAT-567
- Talked and mentor him in person.
- Next up I will interview to collect his thought process.
- ((69c10ce7-08da-41dd-b384-a93312902245))
DONE Ping Anurag via DM
- Had a discussion earlier with Anurag, action items:
- Anurag will ingest the data of 6 countries from 6 RDS databases into Redshift.
- Schema:
airbyte_schema_*(sg, my, nw, etc). - For first step, only need to ingest Retailer tables.
- Schema:
- Me fixes issue “history-only (589 retailers) from different countries”.
- After that we’ll look closer into point 2 and 3.
- Anurag will ingest the data of 6 countries from 6 RDS databases into Redshift.
- Next action from me: waiting for Anurag to ingest data.
- Had a discussion earlier with Anurag, action items:
DONE UIUX learning points from SHUB
DONE Add playwright skills to understand web structure