feat: Cache column offsets for Resource.Get/Set - #2592
Closed
bbernays wants to merge 2 commits into
Closed
Conversation
erezrokah
reviewed
Sep 9, 2026
Member
What's the time we're saving per a regular sync? 30ms? More? |
Contributor
Author
For large plugins like AWS CUR that are not bound by reading from s3, the savings can be significant... assume 100 million rows total... 130 columns per row... If it can save 20 ms per 2,000 rows then with this PR we could see savings 16.66 minutes of CPU time |
erezrokah
reviewed
Sep 9, 2026
Member
There was a problem hiding this comment.
- I would be interested to see the impact of a real sync
- I think we can also avoid caching and pass down the index, e.g.
// scheduler/resolvers/resolvers.go
for _, resource := range resources {
for i := range table.Columns {
resolveColumn(ctx, tableLogger, m, selector, client, resource, i, table.Columns[i], c, classifier)
}
}
func resolveColumn(..., index int, column schema.Column, ...) {
if column.Resolver != nil {
// unchanged — plugin code calls resource.Set(c.Name, v); index cannot reach it
} else {
v := funk.Get(resource.GetItem(), c.ToPascal(column.Name), funk.WithAllowZero())
if v != nil {
if err := resource.SetAtIndex(index, column.Name, v); err != nil { handleErr(err) }
}
}
}
Contributor
Author
|
Closing this in favor of #2593 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During investigation of a plugin with >100 columns in a single table, it was identified that every
Resource.Setcall would iterate through all 100+ columns for each column... This PR introduces the capability to cache column names in a map so that this can be a simple lookupSummary
Resource.GetandResource.Setresolved a column name by scanningTable.Columnslinearly. Resolving a row calls
Setonce per column, so a table with n columns costO(n²) name comparisons per row — and each comparison strides over a full
Columnstruct, so wide tables blow the cache line too.
storeCQIDandStoreCQClientIDeachadded two more scans per row.
This adds an optional
name -> offsetcache onTable:Table.BuildColumnIndex()builds it for a table and its relations.Table.ColumnIndex(name)uses it, falling back toColumns.Index.Scheduler.Syncbuilds it once per table, where the table tree is final.The cache is a hint, never a source of truth.
Table.Columnsis exported andmutated in plenty of places, so a naive map would silently address the wrong column
after a mutation.
ColumnIndexinstead validates every hit againstColumnsbeforereturning it, so a stale cache costs a scan and nothing else. Correctness does not
depend on
BuildColumnIndexever being called, or on invalidation being exhaustive.The mutators that shift offsets (
AddCqIDs,AddCqClientID,OverwriteOrAddColumn,Copy) drop the cache anyway, to keep the fast path fast.Performance
Scheduler sync, 2000 rows, every column set through a resolver (M4 Pro, median of 3).
Benchmark was written to measure this and is not included in the PR:
Narrow tables are unaffected; the crossover against a linear scan is ~25 columns.
Notes for reviewers
schema.Tablenow has an unexported field. Any plugin callingcmp.Diffon aTablewill panic withcannot handle unexported fielduntil it addscmpopts.IgnoreUnexported(schema.Table{}).TestTablesToAndFromArrowneeded exactlythat change. This is the only downstream-visible break and probably deserves a
release note.
Syncnow writes to the tables it is given, so a table tree must not be sharedbetween concurrent
Synccalls.Tables.FilterDfsalready returns copies, so thenormal plugin path is unaffected. Noted at the call site.
resolveColumnalready holds the column offset from itsrange table.Columnsloop and throws it away. Using it would mean exporting aResource.SetAtIndex, i.e. new public API, and the cache recovers most of that winalready. Happy to add it if you'd rather have it.