From 5678ca5ed1d6eda2e405f33d8771c158f25e3050 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuel=20R=C3=BCger?= Date: Mon, 31 Jan 2022 10:57:48 +0100 Subject: [PATCH 01/37] go.mod: Set minimal support version of go to 1.15 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As requested in https://github.com/prometheus/common/pull/353#discussion_r793113705 Signed-off-by: Manuel Rüger --- .circleci/config.yml | 8 -------- README.md | 2 +- go.mod | 2 +- 3 files changed, 2 insertions(+), 10 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 95f2391..6c6f6e4 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -47,14 +47,6 @@ workflows: client_golang: jobs: # Refer to README.md for the currently supported versions. - - test: - name: go-1-13 - go_version: "1.13" - run_lint: true - - test: - name: go-1-14 - go_version: "1.14" - run_lint: true - test: name: go-1-15 go_version: "1.15" diff --git a/README.md b/README.md index 41fc1ed..6b0e874 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ This is the [Go](http://golang.org) client library for instrumenting application code, and one for creating clients that talk to the Prometheus HTTP API. -__This library requires Go1.13 or later.__ +__This library requires Go1.15 or later.__ ## Important note about releases and stability diff --git a/go.mod b/go.mod index 40ff4fc..9088d08 100644 --- a/go.mod +++ b/go.mod @@ -12,4 +12,4 @@ require ( google.golang.org/protobuf v1.26.0 ) -go 1.13 +go 1.15 From f3021b0b81ae2c4adaf201b3e5de6f5f6796d433 Mon Sep 17 00:00:00 2001 From: Kemal Akkoyun Date: Tue, 15 Feb 2022 11:09:48 +0100 Subject: [PATCH 02/37] Create codeql-analysis.yml (#982) Signed-off-by: Kemal Akkoyun --- .github/workflows/codeql-analysis.yml | 70 +++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 .github/workflows/codeql-analysis.yml diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml new file mode 100644 index 0000000..45cf724 --- /dev/null +++ b/.github/workflows/codeql-analysis.yml @@ -0,0 +1,70 @@ +# For most projects, this workflow file will not need changing; you simply need +# to commit it to your repository. +# +# You may wish to alter this file to override the set of languages analyzed, +# or to provide custom queries or build logic. +# +# ******** NOTE ******** +# We have attempted to detect the languages in your repository. Please check +# the `language` matrix defined below to confirm you have the correct set of +# supported CodeQL languages. +# +name: "CodeQL" + +on: + push: + branches: [ main ] + pull_request: + # The branches below must be a subset of the branches above + branches: [ main ] + schedule: + - cron: '31 21 * * 6' + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: [ 'go' ] + # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ] + # Learn more about CodeQL language support at https://git.io/codeql-language-support + + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + # Initializes the CodeQL tools for scanning. + - name: Initialize CodeQL + uses: github/codeql-action/init@v1 + with: + languages: ${{ matrix.language }} + # If you wish to specify custom queries, you can do so here or in a config file. + # By default, queries listed here will override any specified in a config file. + # Prefix the list here with "+" to use these queries and those in the config file. + # queries: ./path/to/local/query, your-org/your-repo/queries@main + + # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). + # If this step fails, then you should remove it and run the build manually (see below) + - name: Autobuild + uses: github/codeql-action/autobuild@v1 + + # ℹ️ Command-line programs to run using the OS shell. + # 📚 https://git.io/JvXDl + + # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines + # and modify them (or add more) to build your code if your project + # uses a compiled language + + #- run: | + # make bootstrap + # make release + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v1 From 1f81b3e9130fe21c55b7db82ecb2d61477358d61 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 23 Feb 2022 12:22:52 +0100 Subject: [PATCH 03/37] Added Transactional Gatherer allowed cached solutions (#989) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Added cached collector. Signed-off-by: Bartlomiej Plotka update. Signed-off-by: Bartlomiej Plotka Attempt 2 Signed-off-by: Bartlomiej Plotka Added blocking registry, with raw collector and transactional handler. Signed-off-by: Bartlomiej Plotka Added fast path to normal (empty) registry to save 8 allocs and 3K5B per Gather. Signed-off-by: Bartlomiej Plotka Simplified API, added tests. Signed-off-by: Bartlomiej Plotka Fix. Signed-off-by: Bartlomiej Plotka Simplified implementation. Signed-off-by: Bartlomiej Plotka Added benchmark. Signed-off-by: Bartlomiej Plotka Optimized. Signed-off-by: Bartlomiej Plotka * Optimization attempt. Signed-off-by: Bartlomiej Plotka * Revert "Optimization attempt." This reverts commit 2fcaf51be9a12b4b95413b6b3e0c13fabfaaf73f. Optimization was not worth it: benchstat v1.txt v2.txt name old time/op new time/op delta CachedTGatherer_Update/Update_of_one_element_without_reset-12 2.64µs ± 0% 4.05µs ± 0% ~ (p=1.000 n=1+1) CachedTGatherer_Update/Update_of_all_elements_with_reset-12 701ms ± 0% 358ms ± 0% ~ (p=1.000 n=1+1) CachedTGatherer_Update/Gather-12 535µs ± 0% 703934µs ± 0% ~ (p=1.000 n=1+1) name old alloc/op new alloc/op delta CachedTGatherer_Update/Update_of_one_element_without_reset-12 208B ± 0% 208B ± 0% ~ (all equal) CachedTGatherer_Update/Update_of_all_elements_with_reset-12 40.2MB ± 0% 41.1MB ± 0% ~ (p=1.000 n=1+1) CachedTGatherer_Update/Gather-12 48.6kB ± 0% 84.3kB ± 0% ~ (p=1.000 n=1+1) name old allocs/op new allocs/op delta CachedTGatherer_Update/Update_of_one_element_without_reset-12 3.00 ± 0% 3.00 ± 0% ~ (all equal) CachedTGatherer_Update/Update_of_all_elements_with_reset-12 6.00 ± 0% 4003.00 ± 0% ~ (p=1.000 n=1+1) CachedTGatherer_Update/Gather-12 1.00k ± 0% 2.01k ± 0% ~ (p=1.000 n=1+1) * nit. Signed-off-by: Bartlomiej Plotka * Another optimization attempt. Signed-off-by: Bartlomiej Plotka * rename and further optimization. Signed-off-by: Bartlomiej Plotka * Hopefully final optimization. benchstat -delta-test=none v6.txt v9.txt name old time/op new time/op delta CachedTGatherer_Update/Update_of_one_element_without_reset-12 13.1ms ± 0% 0.0ms ± 0% -99.81% CachedTGatherer_Update/Update_of_all_elements_with_reset-12 309ms ± 0% 282ms ± 0% -8.77% CachedTGatherer_Update/Gather-12 422ms ± 0% 0ms ± 0% -99.95% name old alloc/op new alloc/op delta CachedTGatherer_Update/Update_of_one_element_without_reset-12 208B ± 0% 208B ± 0% 0.00% CachedTGatherer_Update/Update_of_all_elements_with_reset-12 2.47kB ± 0% 1.67kB ± 0% -32.56% CachedTGatherer_Update/Gather-12 52.8kB ± 0% 24.6kB ± 0% -53.34% name old allocs/op new allocs/op delta CachedTGatherer_Update/Update_of_one_element_without_reset-12 3.00 ± 0% 3.00 ± 0% 0.00% CachedTGatherer_Update/Update_of_all_elements_with_reset-12 0.00 0.00 0.00% CachedTGatherer_Update/Gather-12 1.00k ± 0% 0.00k ± 0% -99.60% Signed-off-by: Bartlomiej Plotka * Removed obsolete comment Signed-off-by: Bartlomiej Plotka * Fixed tests. Signed-off-by: Bartlomiej Plotka * Removed cache. Signed-off-by: Bartlomiej Plotka * Fixed tests. Signed-off-by: Bartlomiej Plotka * Re-add cache. Signed-off-by: Bartlomiej Plotka * Removed cache. Signed-off-by: Bartlomiej Plotka --- prometheus/desc.go | 4 +- prometheus/internal/metric.go | 28 ++++++-- prometheus/metric.go | 16 ----- prometheus/promhttp/http.go | 10 ++- prometheus/promhttp/http_test.go | 107 ++++++++++++++++++++++++------- prometheus/registry.go | 101 +++++++++++++++++++++++++++-- prometheus/registry_test.go | 80 +++++++++++++++++++++++ prometheus/testutil/testutil.go | 11 +++- prometheus/value.go | 44 ++++++++++--- prometheus/wrap.go | 3 +- 10 files changed, 340 insertions(+), 64 deletions(-) diff --git a/prometheus/desc.go b/prometheus/desc.go index 4bb816a..ee81107 100644 --- a/prometheus/desc.go +++ b/prometheus/desc.go @@ -20,6 +20,8 @@ import ( "strings" "github.com/cespare/xxhash/v2" + "github.com/prometheus/client_golang/prometheus/internal" + //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility. "github.com/golang/protobuf/proto" "github.com/prometheus/common/model" @@ -154,7 +156,7 @@ func NewDesc(fqName, help string, variableLabels []string, constLabels Labels) * Value: proto.String(v), }) } - sort.Sort(labelPairSorter(d.constLabelPairs)) + sort.Sort(internal.LabelPairSorter(d.constLabelPairs)) return d } diff --git a/prometheus/internal/metric.go b/prometheus/internal/metric.go index 351c26e..6515c11 100644 --- a/prometheus/internal/metric.go +++ b/prometheus/internal/metric.go @@ -19,18 +19,34 @@ import ( dto "github.com/prometheus/client_model/go" ) -// metricSorter is a sortable slice of *dto.Metric. -type metricSorter []*dto.Metric +// LabelPairSorter implements sort.Interface. It is used to sort a slice of +// dto.LabelPair pointers. +type LabelPairSorter []*dto.LabelPair -func (s metricSorter) Len() int { +func (s LabelPairSorter) Len() int { return len(s) } -func (s metricSorter) Swap(i, j int) { +func (s LabelPairSorter) Swap(i, j int) { s[i], s[j] = s[j], s[i] } -func (s metricSorter) Less(i, j int) bool { +func (s LabelPairSorter) Less(i, j int) bool { + return s[i].GetName() < s[j].GetName() +} + +// MetricSorter is a sortable slice of *dto.Metric. +type MetricSorter []*dto.Metric + +func (s MetricSorter) Len() int { + return len(s) +} + +func (s MetricSorter) Swap(i, j int) { + s[i], s[j] = s[j], s[i] +} + +func (s MetricSorter) Less(i, j int) bool { if len(s[i].Label) != len(s[j].Label) { // This should not happen. The metrics are // inconsistent. However, we have to deal with the fact, as @@ -68,7 +84,7 @@ func (s metricSorter) Less(i, j int) bool { // the slice, with the contained Metrics sorted within each MetricFamily. func NormalizeMetricFamilies(metricFamiliesByName map[string]*dto.MetricFamily) []*dto.MetricFamily { for _, mf := range metricFamiliesByName { - sort.Sort(metricSorter(mf.Metric)) + sort.Sort(MetricSorter(mf.Metric)) } names := make([]string, 0, len(metricFamiliesByName)) for name, mf := range metricFamiliesByName { diff --git a/prometheus/metric.go b/prometheus/metric.go index dc12191..118a54e 100644 --- a/prometheus/metric.go +++ b/prometheus/metric.go @@ -115,22 +115,6 @@ func BuildFQName(namespace, subsystem, name string) string { return name } -// labelPairSorter implements sort.Interface. It is used to sort a slice of -// dto.LabelPair pointers. -type labelPairSorter []*dto.LabelPair - -func (s labelPairSorter) Len() int { - return len(s) -} - -func (s labelPairSorter) Swap(i, j int) { - s[i], s[j] = s[j], s[i] -} - -func (s labelPairSorter) Less(i, j int) bool { - return s[i].GetName() < s[j].GetName() -} - type invalidMetric struct { desc *Desc err error diff --git a/prometheus/promhttp/http.go b/prometheus/promhttp/http.go index d86d0cf..a6e4f85 100644 --- a/prometheus/promhttp/http.go +++ b/prometheus/promhttp/http.go @@ -84,6 +84,13 @@ func Handler() http.Handler { // instrumentation. Use the InstrumentMetricHandler function to apply the same // kind of instrumentation as it is used by the Handler function. func HandlerFor(reg prometheus.Gatherer, opts HandlerOpts) http.Handler { + return HandlerForTransactional(prometheus.ToTransactionalGatherer(reg), opts) +} + +// HandlerForTransactional is like HandlerFor, but it uses transactional gather, which +// can safely change in-place returned *dto.MetricFamily before call to `Gather` and after +// call to `done` of that `Gather`. +func HandlerForTransactional(reg prometheus.TransactionalGatherer, opts HandlerOpts) http.Handler { var ( inFlightSem chan struct{} errCnt = prometheus.NewCounterVec( @@ -123,7 +130,8 @@ func HandlerFor(reg prometheus.Gatherer, opts HandlerOpts) http.Handler { return } } - mfs, err := reg.Gather() + mfs, done, err := reg.Gather() + defer done() if err != nil { if opts.ErrorLog != nil { opts.ErrorLog.Println("error gathering metrics:", err) diff --git a/prometheus/promhttp/http_test.go b/prometheus/promhttp/http_test.go index 781ea8f..53204c5 100644 --- a/prometheus/promhttp/http_test.go +++ b/prometheus/promhttp/http_test.go @@ -16,6 +16,7 @@ package promhttp import ( "bytes" "errors" + "fmt" "log" "net/http" "net/http/httptest" @@ -24,6 +25,7 @@ import ( "time" "github.com/prometheus/client_golang/prometheus" + dto "github.com/prometheus/client_model/go" ) type errorCollector struct{} @@ -56,8 +58,19 @@ func (b blockingCollector) Collect(ch chan<- prometheus.Metric) { <-b.Block } -func TestHandlerErrorHandling(t *testing.T) { +type mockTransactionGatherer struct { + g prometheus.Gatherer + gatherInvoked int + doneInvoked int +} +func (g *mockTransactionGatherer) Gather() (_ []*dto.MetricFamily, done func(), err error) { + g.gatherInvoked++ + mfs, err := g.g.Gather() + return mfs, func() { g.doneInvoked++ }, err +} + +func TestHandlerErrorHandling(t *testing.T) { // Create a registry that collects a MetricFamily with two elements, // another with one, and reports an error. Further down, we'll use the // same registry in the HandlerOpts. @@ -90,21 +103,30 @@ func TestHandlerErrorHandling(t *testing.T) { request, _ := http.NewRequest("GET", "/", nil) request.Header.Add("Accept", "test/plain") - errorHandler := HandlerFor(reg, HandlerOpts{ + mReg := &mockTransactionGatherer{g: reg} + errorHandler := HandlerForTransactional(mReg, HandlerOpts{ ErrorLog: logger, ErrorHandling: HTTPErrorOnError, Registry: reg, }) - continueHandler := HandlerFor(reg, HandlerOpts{ + continueHandler := HandlerForTransactional(mReg, HandlerOpts{ ErrorLog: logger, ErrorHandling: ContinueOnError, Registry: reg, }) - panicHandler := HandlerFor(reg, HandlerOpts{ + panicHandler := HandlerForTransactional(mReg, HandlerOpts{ ErrorLog: logger, ErrorHandling: PanicOnError, Registry: reg, }) + // Expect gatherer not touched. + if got := mReg.gatherInvoked; got != 0 { + t.Fatalf("unexpected number of gather invokes, want 0, got %d", got) + } + if got := mReg.doneInvoked; got != 0 { + t.Fatalf("unexpected number of done invokes, want 0, got %d", got) + } + wantMsg := `error gathering metrics: error collecting metric Desc{fqName: "invalid_metric", help: "not helpful", constLabels: {}, variableLabels: []}: collect error ` wantErrorBody := `An error has occurred while serving metrics: @@ -140,25 +162,39 @@ the_count 0 ` errorHandler.ServeHTTP(writer, request) + if got := mReg.gatherInvoked; got != 1 { + t.Fatalf("unexpected number of gather invokes, want 1, got %d", got) + } + if got := mReg.doneInvoked; got != 1 { + t.Fatalf("unexpected number of done invokes, want 1, got %d", got) + } if got, want := writer.Code, http.StatusInternalServerError; got != want { t.Errorf("got HTTP status code %d, want %d", got, want) } - if got := logBuf.String(); got != wantMsg { - t.Errorf("got log message:\n%s\nwant log message:\n%s\n", got, wantMsg) + if got, want := logBuf.String(), wantMsg; got != want { + t.Errorf("got log buf %q, want %q", got, want) } - if got := writer.Body.String(); got != wantErrorBody { - t.Errorf("got body:\n%s\nwant body:\n%s\n", got, wantErrorBody) + if got, want := writer.Body.String(), wantErrorBody; got != want { + t.Errorf("got body %q, want %q", got, want) } + logBuf.Reset() writer.Body.Reset() writer.Code = http.StatusOK continueHandler.ServeHTTP(writer, request) + + if got := mReg.gatherInvoked; got != 2 { + t.Fatalf("unexpected number of gather invokes, want 2, got %d", got) + } + if got := mReg.doneInvoked; got != 2 { + t.Fatalf("unexpected number of done invokes, want 2, got %d", got) + } if got, want := writer.Code, http.StatusOK; got != want { t.Errorf("got HTTP status code %d, want %d", got, want) } - if got := logBuf.String(); got != wantMsg { - t.Errorf("got log message %q, want %q", got, wantMsg) + if got, want := logBuf.String(), wantMsg; got != want { + t.Errorf("got log buf %q, want %q", got, want) } if got := writer.Body.String(); got != wantOKBody1 && got != wantOKBody2 { t.Errorf("got body %q, want either %q or %q", got, wantOKBody1, wantOKBody2) @@ -168,20 +204,34 @@ the_count 0 if err := recover(); err == nil { t.Error("expected panic from panicHandler") } + if got := mReg.gatherInvoked; got != 3 { + t.Fatalf("unexpected number of gather invokes, want 3, got %d", got) + } + if got := mReg.doneInvoked; got != 3 { + t.Fatalf("unexpected number of done invokes, want 3, got %d", got) + } }() panicHandler.ServeHTTP(writer, request) } func TestInstrumentMetricHandler(t *testing.T) { reg := prometheus.NewRegistry() - handler := InstrumentMetricHandler(reg, HandlerFor(reg, HandlerOpts{})) + mReg := &mockTransactionGatherer{g: reg} + handler := InstrumentMetricHandler(reg, HandlerForTransactional(mReg, HandlerOpts{})) // Do it again to test idempotency. - InstrumentMetricHandler(reg, HandlerFor(reg, HandlerOpts{})) + InstrumentMetricHandler(reg, HandlerForTransactional(mReg, HandlerOpts{})) writer := httptest.NewRecorder() request, _ := http.NewRequest("GET", "/", nil) request.Header.Add("Accept", "test/plain") handler.ServeHTTP(writer, request) + if got := mReg.gatherInvoked; got != 1 { + t.Fatalf("unexpected number of gather invokes, want 1, got %d", got) + } + if got := mReg.doneInvoked; got != 1 { + t.Fatalf("unexpected number of done invokes, want 1, got %d", got) + } + if got, want := writer.Code, http.StatusOK; got != want { t.Errorf("got HTTP status code %d, want %d", got, want) } @@ -195,19 +245,28 @@ func TestInstrumentMetricHandler(t *testing.T) { t.Errorf("got body %q, does not contain %q", got, want) } - writer.Body.Reset() - handler.ServeHTTP(writer, request) - if got, want := writer.Code, http.StatusOK; got != want { - t.Errorf("got HTTP status code %d, want %d", got, want) - } + for i := 0; i < 100; i++ { + writer.Body.Reset() + handler.ServeHTTP(writer, request) - want = "promhttp_metric_handler_requests_in_flight 1\n" - if got := writer.Body.String(); !strings.Contains(got, want) { - t.Errorf("got body %q, does not contain %q", got, want) - } - want = "promhttp_metric_handler_requests_total{code=\"200\"} 1\n" - if got := writer.Body.String(); !strings.Contains(got, want) { - t.Errorf("got body %q, does not contain %q", got, want) + if got, want := mReg.gatherInvoked, i+2; got != want { + t.Fatalf("unexpected number of gather invokes, want %d, got %d", want, got) + } + if got, want := mReg.doneInvoked, i+2; got != want { + t.Fatalf("unexpected number of done invokes, want %d, got %d", want, got) + } + if got, want := writer.Code, http.StatusOK; got != want { + t.Errorf("got HTTP status code %d, want %d", got, want) + } + + want := "promhttp_metric_handler_requests_in_flight 1\n" + if got := writer.Body.String(); !strings.Contains(got, want) { + t.Errorf("got body %q, does not contain %q", got, want) + } + want = fmt.Sprintf("promhttp_metric_handler_requests_total{code=\"200\"} %d\n", i+1) + if got := writer.Body.String(); !strings.Contains(got, want) { + t.Errorf("got body %q, does not contain %q", got, want) + } } } diff --git a/prometheus/registry.go b/prometheus/registry.go index 383a7f5..5046f7e 100644 --- a/prometheus/registry.go +++ b/prometheus/registry.go @@ -407,6 +407,14 @@ func (r *Registry) MustRegister(cs ...Collector) { // Gather implements Gatherer. func (r *Registry) Gather() ([]*dto.MetricFamily, error) { + r.mtx.RLock() + + if len(r.collectorsByID) == 0 && len(r.uncheckedCollectors) == 0 { + // Fast path. + r.mtx.RUnlock() + return nil, nil + } + var ( checkedMetricChan = make(chan Metric, capMetricChan) uncheckedMetricChan = make(chan Metric, capMetricChan) @@ -416,7 +424,6 @@ func (r *Registry) Gather() ([]*dto.MetricFamily, error) { registeredDescIDs map[uint64]struct{} // Only used for pedantic checks ) - r.mtx.RLock() goroutineBudget := len(r.collectorsByID) + len(r.uncheckedCollectors) metricFamiliesByName := make(map[string]*dto.MetricFamily, len(r.dimHashesByName)) checkedCollectors := make(chan Collector, len(r.collectorsByID)) @@ -884,11 +891,11 @@ func checkMetricConsistency( h.Write(separatorByteSlice) // Make sure label pairs are sorted. We depend on it for the consistency // check. - if !sort.IsSorted(labelPairSorter(dtoMetric.Label)) { + if !sort.IsSorted(internal.LabelPairSorter(dtoMetric.Label)) { // We cannot sort dtoMetric.Label in place as it is immutable by contract. copiedLabels := make([]*dto.LabelPair, len(dtoMetric.Label)) copy(copiedLabels, dtoMetric.Label) - sort.Sort(labelPairSorter(copiedLabels)) + sort.Sort(internal.LabelPairSorter(copiedLabels)) dtoMetric.Label = copiedLabels } for _, lp := range dtoMetric.Label { @@ -935,7 +942,7 @@ func checkDescConsistency( metricFamily.GetName(), dtoMetric, desc, ) } - sort.Sort(labelPairSorter(lpsFromDesc)) + sort.Sort(internal.LabelPairSorter(lpsFromDesc)) for i, lpFromDesc := range lpsFromDesc { lpFromMetric := dtoMetric.Label[i] if lpFromDesc.GetName() != lpFromMetric.GetName() || @@ -948,3 +955,89 @@ func checkDescConsistency( } return nil } + +var _ TransactionalGatherer = &MultiTRegistry{} + +// MultiTRegistry is a TransactionalGatherer that joins gathered metrics from multiple +// transactional gatherers. +// +// It is caller responsibility to ensure two registries have mutually exclusive metric families, +// no deduplication will happen. +type MultiTRegistry struct { + tGatherers []TransactionalGatherer +} + +// NewMultiTRegistry creates MultiTRegistry. +func NewMultiTRegistry(tGatherers ...TransactionalGatherer) *MultiTRegistry { + return &MultiTRegistry{ + tGatherers: tGatherers, + } +} + +// Gather implements TransactionalGatherer interface. +func (r *MultiTRegistry) Gather() (mfs []*dto.MetricFamily, done func(), err error) { + errs := MultiError{} + + dFns := make([]func(), 0, len(r.tGatherers)) + // TODO(bwplotka): Implement concurrency for those? + for _, g := range r.tGatherers { + // TODO(bwplotka): Check for duplicates? + m, d, err := g.Gather() + errs.Append(err) + + mfs = append(mfs, m...) + dFns = append(dFns, d) + } + + // TODO(bwplotka): Consider sort in place, given metric family in gather is sorted already. + sort.Slice(mfs, func(i, j int) bool { + return *mfs[i].Name < *mfs[j].Name + }) + return mfs, func() { + for _, d := range dFns { + d() + } + }, errs.MaybeUnwrap() +} + +// TransactionalGatherer represents transactional gatherer that can be triggered to notify gatherer that memory +// used by metric family is no longer used by a caller. This allows implementations with cache. +type TransactionalGatherer interface { + // Gather returns metrics in a lexicographically sorted slice + // of uniquely named MetricFamily protobufs. Gather ensures that the + // returned slice is valid and self-consistent so that it can be used + // for valid exposition. As an exception to the strict consistency + // requirements described for metric.Desc, Gather will tolerate + // different sets of label names for metrics of the same metric family. + // + // Even if an error occurs, Gather attempts to gather as many metrics as + // possible. Hence, if a non-nil error is returned, the returned + // MetricFamily slice could be nil (in case of a fatal error that + // prevented any meaningful metric collection) or contain a number of + // MetricFamily protobufs, some of which might be incomplete, and some + // might be missing altogether. The returned error (which might be a + // MultiError) explains the details. Note that this is mostly useful for + // debugging purposes. If the gathered protobufs are to be used for + // exposition in actual monitoring, it is almost always better to not + // expose an incomplete result and instead disregard the returned + // MetricFamily protobufs in case the returned error is non-nil. + // + // Important: done is expected to be triggered (even if the error occurs!) + // once caller does not need returned slice of dto.MetricFamily. + Gather() (_ []*dto.MetricFamily, done func(), err error) +} + +// ToTransactionalGatherer transforms Gatherer to transactional one with noop as done function. +func ToTransactionalGatherer(g Gatherer) TransactionalGatherer { + return &noTransactionGatherer{g: g} +} + +type noTransactionGatherer struct { + g Gatherer +} + +// Gather implements TransactionalGatherer interface. +func (g *noTransactionGatherer) Gather() (_ []*dto.MetricFamily, done func(), err error) { + mfs, err := g.g.Gather() + return mfs, func() {}, err +} diff --git a/prometheus/registry_test.go b/prometheus/registry_test.go index 0ff7a64..7a959da 100644 --- a/prometheus/registry_test.go +++ b/prometheus/registry_test.go @@ -21,6 +21,7 @@ package prometheus_test import ( "bytes" + "errors" "fmt" "io/ioutil" "math/rand" @@ -1175,3 +1176,82 @@ func TestAlreadyRegisteredCollision(t *testing.T) { } } } + +type tGatherer struct { + done bool + err error +} + +func (g *tGatherer) Gather() (_ []*dto.MetricFamily, done func(), err error) { + name := "g1" + val := 1.0 + return []*dto.MetricFamily{ + {Name: &name, Metric: []*dto.Metric{{Gauge: &dto.Gauge{Value: &val}}}}, + }, func() { g.done = true }, g.err +} + +func TestNewMultiTRegistry(t *testing.T) { + treg := &tGatherer{} + + t.Run("one registry", func(t *testing.T) { + m := prometheus.NewMultiTRegistry(treg) + ret, done, err := m.Gather() + if err != nil { + t.Error("gather failed:", err) + } + done() + if len(ret) != 1 { + t.Error("unexpected number of metric families, expected 1, got", ret) + } + if !treg.done { + t.Error("inner transactional registry not marked as done") + } + }) + + reg := prometheus.NewRegistry() + if err := reg.Register(prometheus.NewCounter(prometheus.CounterOpts{Name: "c1", Help: "help c1"})); err != nil { + t.Error("registration failed:", err) + } + + // Note on purpose two registries will have exactly same metric family name (but with different string). + // This behaviour is undefined at the moment. + if err := reg.Register(prometheus.NewGauge(prometheus.GaugeOpts{Name: "g1", Help: "help g1"})); err != nil { + t.Error("registration failed:", err) + } + treg.done = false + + t.Run("two registries", func(t *testing.T) { + m := prometheus.NewMultiTRegistry(prometheus.ToTransactionalGatherer(reg), treg) + ret, done, err := m.Gather() + if err != nil { + t.Error("gather failed:", err) + } + done() + if len(ret) != 3 { + t.Error("unexpected number of metric families, expected 3, got", ret) + } + if !treg.done { + t.Error("inner transactional registry not marked as done") + } + }) + + treg.done = false + // Inject error. + treg.err = errors.New("test err") + + t.Run("two registries, one with error", func(t *testing.T) { + m := prometheus.NewMultiTRegistry(prometheus.ToTransactionalGatherer(reg), treg) + ret, done, err := m.Gather() + if err != treg.err { + t.Error("unexpected error:", err) + } + done() + if len(ret) != 3 { + t.Error("unexpected number of metric families, expected 3, got", ret) + } + // Still on error, we expect done to be triggered. + if !treg.done { + t.Error("inner transactional registry not marked as done") + } + }) +} diff --git a/prometheus/testutil/testutil.go b/prometheus/testutil/testutil.go index 9af60ce..bf95bea 100644 --- a/prometheus/testutil/testutil.go +++ b/prometheus/testutil/testutil.go @@ -167,7 +167,16 @@ func CollectAndCompare(c prometheus.Collector, expected io.Reader, metricNames . // exposition format. If any metricNames are provided, only metrics with those // names are compared. func GatherAndCompare(g prometheus.Gatherer, expected io.Reader, metricNames ...string) error { - got, err := g.Gather() + return TransactionalGatherAndCompare(prometheus.ToTransactionalGatherer(g), expected, metricNames...) +} + +// TransactionalGatherAndCompare gathers all metrics from the provided Gatherer and compares +// it to an expected output read from the provided Reader in the Prometheus text +// exposition format. If any metricNames are provided, only metrics with those +// names are compared. +func TransactionalGatherAndCompare(g prometheus.TransactionalGatherer, expected io.Reader, metricNames ...string) error { + got, done, err := g.Gather() + defer done() if err != nil { return fmt.Errorf("gathering metrics failed: %s", err) } diff --git a/prometheus/value.go b/prometheus/value.go index b4e0ae1..9f10695 100644 --- a/prometheus/value.go +++ b/prometheus/value.go @@ -21,6 +21,7 @@ import ( //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility. "github.com/golang/protobuf/proto" + "github.com/prometheus/client_golang/prometheus/internal" "google.golang.org/protobuf/types/known/timestamppb" dto "github.com/prometheus/client_model/go" @@ -38,6 +39,23 @@ const ( UntypedValue ) +var ( + CounterMetricTypePtr = func() *dto.MetricType { d := dto.MetricType_COUNTER; return &d }() + GaugeMetricTypePtr = func() *dto.MetricType { d := dto.MetricType_GAUGE; return &d }() + UntypedMetricTypePtr = func() *dto.MetricType { d := dto.MetricType_UNTYPED; return &d }() +) + +func (v ValueType) ToDTO() *dto.MetricType { + switch v { + case CounterValue: + return CounterMetricTypePtr + case GaugeValue: + return GaugeMetricTypePtr + default: + return UntypedMetricTypePtr + } +} + // valueFunc is a generic metric for simple values retrieved on collect time // from a function. It implements Metric and Collector. Its effective type is // determined by ValueType. This is a low-level building block used by the @@ -91,11 +109,15 @@ func NewConstMetric(desc *Desc, valueType ValueType, value float64, labelValues if err := validateLabelValues(labelValues, len(desc.variableLabels)); err != nil { return nil, err } + + metric := &dto.Metric{} + if err := populateMetric(valueType, value, MakeLabelPairs(desc, labelValues), nil, metric); err != nil { + return nil, err + } + return &constMetric{ - desc: desc, - valType: valueType, - val: value, - labelPairs: MakeLabelPairs(desc, labelValues), + desc: desc, + metric: metric, }, nil } @@ -110,10 +132,8 @@ func MustNewConstMetric(desc *Desc, valueType ValueType, value float64, labelVal } type constMetric struct { - desc *Desc - valType ValueType - val float64 - labelPairs []*dto.LabelPair + desc *Desc + metric *dto.Metric } func (m *constMetric) Desc() *Desc { @@ -121,7 +141,11 @@ func (m *constMetric) Desc() *Desc { } func (m *constMetric) Write(out *dto.Metric) error { - return populateMetric(m.valType, m.val, m.labelPairs, nil, out) + out.Label = m.metric.Label + out.Counter = m.metric.Counter + out.Gauge = m.metric.Gauge + out.Untyped = m.metric.Untyped + return nil } func populateMetric( @@ -170,7 +194,7 @@ func MakeLabelPairs(desc *Desc, labelValues []string) []*dto.LabelPair { }) } labelPairs = append(labelPairs, desc.constLabelPairs...) - sort.Sort(labelPairSorter(labelPairs)) + sort.Sort(internal.LabelPairSorter(labelPairs)) return labelPairs } diff --git a/prometheus/wrap.go b/prometheus/wrap.go index 74ee932..c29f94b 100644 --- a/prometheus/wrap.go +++ b/prometheus/wrap.go @@ -20,6 +20,7 @@ import ( //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility. "github.com/golang/protobuf/proto" + "github.com/prometheus/client_golang/prometheus/internal" dto "github.com/prometheus/client_model/go" ) @@ -182,7 +183,7 @@ func (m *wrappingMetric) Write(out *dto.Metric) error { Value: proto.String(lv), }) } - sort.Sort(labelPairSorter(out.Label)) + sort.Sort(internal.LabelPairSorter(out.Label)) return nil } From 868ec2137f6748fea08e345af668655c4c9e26ce Mon Sep 17 00:00:00 2001 From: PrometheusBot Date: Wed, 16 Mar 2022 09:26:54 +0100 Subject: [PATCH 04/37] Update common Prometheus files (#992) Signed-off-by: prombot --- .github/workflows/golangci-lint.yml | 4 ++-- Makefile.common | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index f96c76a..99a9ab4 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -23,9 +23,9 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Lint uses: golangci/golangci-lint-action@v2 with: - version: v1.42.0 + version: v1.44.2 diff --git a/Makefile.common b/Makefile.common index bf7304e..5ab1b14 100644 --- a/Makefile.common +++ b/Makefile.common @@ -83,7 +83,7 @@ PROMU_URL := https://github.com/prometheus/promu/releases/download/v$(PROMU_ GOLANGCI_LINT := GOLANGCI_LINT_OPTS ?= -GOLANGCI_LINT_VERSION ?= v1.42.0 +GOLANGCI_LINT_VERSION ?= v1.44.2 # golangci-lint only supports linux, darwin and windows platforms on i386/amd64. # windows isn't included here because of the path separator being different. ifeq ($(GOHOSTOS),$(filter $(GOHOSTOS),linux darwin)) From b05177a55370cd3f5e11450619a31b3ccbfa2e2b Mon Sep 17 00:00:00 2001 From: alissa-tung Date: Wed, 16 Mar 2022 17:46:48 +0800 Subject: [PATCH 05/37] Fix deprecated `NewBuildInfoCollector` API Update `examples/random/main.go`: `prometheus.NewBuildInfoCollector` is deprecated. Use `collectors.NewBuildInfoCollector` instead. Signed-off-by: alissa-tung --- examples/random/main.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/examples/random/main.go b/examples/random/main.go index cf44863..1321423 100644 --- a/examples/random/main.go +++ b/examples/random/main.go @@ -26,6 +26,7 @@ import ( "time" "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" "github.com/prometheus/client_golang/prometheus/promhttp" ) @@ -67,7 +68,7 @@ func main() { prometheus.MustRegister(rpcDurations) prometheus.MustRegister(rpcDurationsHistogram) // Add Go module build info. - prometheus.MustRegister(prometheus.NewBuildInfoCollector()) + prometheus.MustRegister(collectors.NewBuildInfoCollector()) start := time.Now() From ffd6362a06df307874df9230b3e9394d9b037523 Mon Sep 17 00:00:00 2001 From: alissa-tung Date: Wed, 16 Mar 2022 18:14:51 +0800 Subject: [PATCH 06/37] Update Dockerfile example Signed-off-by: alissa-tung Co-authored-by: Cruise_Hua --- Dockerfile | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 4da5f16..2627ff4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,15 @@ # This Dockerfile builds an image for a client_golang example. # # Use as (from the root for the client_golang repository): -# docker build -f examples/$name/Dockerfile -t prometheus/golang-example-$name . +# docker build -f Dockerfile -t prometheus/golang-example . + +# Run as +# docker run -P prometheus/golang-example /random +# or +# docker run -P prometheus/golang-example /simple + +# Test as +# curl $ip:$port/metrics # Builder image, where we build the example. FROM golang:1 AS builder From 0291563b9fcfe1b7b272ca040456ee137fbbb33d Mon Sep 17 00:00:00 2001 From: prombot Date: Wed, 16 Mar 2022 19:50:22 +0000 Subject: [PATCH 07/37] Update common Prometheus files Signed-off-by: prombot --- .circleci/config.yml | 2 +- .github/workflows/golangci-lint.yml | 14 +++++--------- 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 6c6f6e4..9bb9886 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -1,7 +1,7 @@ version: 2.1 orbs: go: circleci/go@0.2.0 - prometheus: prometheus/prometheus@0.15.0 + prometheus: prometheus/prometheus@0.16.0 jobs: test: parameters: diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index 99a9ab4..8211b9a 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -9,13 +9,6 @@ on: - ".github/workflows/golangci-lint.yml" - ".golangci.yml" pull_request: - paths: - - "go.sum" - - "go.mod" - - "**.go" - - "scripts/errcheck_excludes.txt" - - ".github/workflows/golangci-lint.yml" - - ".golangci.yml" jobs: golangci: @@ -24,8 +17,11 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@v3 - + - name: install Go + uses: actions/setup-go@v2 + with: + go-version: 1.17.x - name: Lint - uses: golangci/golangci-lint-action@v2 + uses: golangci/golangci-lint-action@v3.1.0 with: version: v1.44.2 From 66837e3298bdc57a828794c23bacb253ed8c04cd Mon Sep 17 00:00:00 2001 From: William Perron Date: Thu, 17 Mar 2022 07:30:41 -0400 Subject: [PATCH 08/37] Add exemplar support for const histogram and const metric (#986) * Add support for exemplars on constHistogram Co-authored-by: William Perron Signed-off-by: William Perron * remove GetExemplars function Signed-off-by: William Perron * fixed linting warnings reduce repetition in constHistogram w/ exemplar Signed-off-by: William Perron * Add values to correct bucket Signed-off-by: William Perron * Misc fixes Co-authored-by: Francis Bogsanyi Signed-off-by: William Perron * avoid panic when there are fewer buckets than exemplars Co-authored-by: Arun Mahendra Signed-off-by: William Perron * Added MustNewMetricWithExemplars that wraps metrics with exemplar (#3) Changes: * Make sure to not "leak" dto.Metric * Reused upper bounds we already have for histogram * Common code for all types. Signed-off-by: Bartlomiej Plotka Co-authored-by: Arun Mahendra Co-authored-by: Bartlomiej Plotka --- prometheus/examples_test.go | 112 ++++++++++++++++++++++++++++++++++- prometheus/histogram.go | 2 +- prometheus/histogram_test.go | 12 ++-- prometheus/metric.go | 90 ++++++++++++++++++++++++++++ prometheus/metric_test.go | 45 +++++++++++++- 5 files changed, 251 insertions(+), 10 deletions(-) diff --git a/prometheus/examples_test.go b/prometheus/examples_test.go index a73ed18..a7a8d0c 100644 --- a/prometheus/examples_test.go +++ b/prometheus/examples_test.go @@ -24,9 +24,8 @@ import ( //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility. "github.com/golang/protobuf/proto" - "github.com/prometheus/common/expfmt" - dto "github.com/prometheus/client_model/go" + "github.com/prometheus/common/expfmt" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" @@ -599,6 +598,115 @@ func ExampleNewConstHistogram() { // > } +func ExampleNewConstHistogram_WithExemplar() { + desc := prometheus.NewDesc( + "http_request_duration_seconds", + "A histogram of the HTTP request durations.", + []string{"code", "method"}, + prometheus.Labels{"owner": "example"}, + ) + + // Create a constant histogram from values we got from a 3rd party telemetry system. + h := prometheus.MustNewConstHistogram( + desc, + 4711, 403.34, + map[float64]uint64{25: 121, 50: 2403, 100: 3221, 200: 4233}, + "200", "get", + ) + + // Wrap const histogram with exemplars for each bucket. + exemplarTs, _ := time.Parse(time.RFC850, "Monday, 02-Jan-06 15:04:05 GMT") + exemplarLabels := prometheus.Labels{"testName": "testVal"} + h = prometheus.MustNewMetricWithExemplars( + h, + prometheus.Exemplar{Labels: exemplarLabels, Timestamp: exemplarTs, Value: 24.0}, + prometheus.Exemplar{Labels: exemplarLabels, Timestamp: exemplarTs, Value: 42.0}, + prometheus.Exemplar{Labels: exemplarLabels, Timestamp: exemplarTs, Value: 89.0}, + prometheus.Exemplar{Labels: exemplarLabels, Timestamp: exemplarTs, Value: 157.0}, + ) + + // Just for demonstration, let's check the state of the histogram by + // (ab)using its Write method (which is usually only used by Prometheus + // internally). + metric := &dto.Metric{} + h.Write(metric) + fmt.Println(proto.MarshalTextString(metric)) + + // Output: + // label: < + // name: "code" + // value: "200" + // > + // label: < + // name: "method" + // value: "get" + // > + // label: < + // name: "owner" + // value: "example" + // > + // histogram: < + // sample_count: 4711 + // sample_sum: 403.34 + // bucket: < + // cumulative_count: 121 + // upper_bound: 25 + // exemplar: < + // label: < + // name: "testName" + // value: "testVal" + // > + // value: 24 + // timestamp: < + // seconds: 1136214245 + // > + // > + // > + // bucket: < + // cumulative_count: 2403 + // upper_bound: 50 + // exemplar: < + // label: < + // name: "testName" + // value: "testVal" + // > + // value: 42 + // timestamp: < + // seconds: 1136214245 + // > + // > + // > + // bucket: < + // cumulative_count: 3221 + // upper_bound: 100 + // exemplar: < + // label: < + // name: "testName" + // value: "testVal" + // > + // value: 89 + // timestamp: < + // seconds: 1136214245 + // > + // > + // > + // bucket: < + // cumulative_count: 4233 + // upper_bound: 200 + // exemplar: < + // label: < + // name: "testName" + // value: "testVal" + // > + // value: 157 + // timestamp: < + // seconds: 1136214245 + // > + // > + // > + // > +} + func ExampleAlreadyRegisteredError() { reqCounter := prometheus.NewCounter(prometheus.CounterOpts{ Name: "requests_total", diff --git a/prometheus/histogram.go b/prometheus/histogram.go index 893802f..0d47fec 100644 --- a/prometheus/histogram.go +++ b/prometheus/histogram.go @@ -581,11 +581,11 @@ func (h *constHistogram) Desc() *Desc { func (h *constHistogram) Write(out *dto.Metric) error { his := &dto.Histogram{} + buckets := make([]*dto.Bucket, 0, len(h.buckets)) his.SampleCount = proto.Uint64(h.count) his.SampleSum = proto.Float64(h.sum) - for upperBound, count := range h.buckets { buckets = append(buckets, &dto.Bucket{ CumulativeCount: proto.Uint64(count), diff --git a/prometheus/histogram_test.go b/prometheus/histogram_test.go index da57715..b96eff9 100644 --- a/prometheus/histogram_test.go +++ b/prometheus/histogram_test.go @@ -424,24 +424,24 @@ func TestHistogramExemplar(t *testing.T) { } expectedExemplars := []*dto.Exemplar{ nil, - &dto.Exemplar{ + { Label: []*dto.LabelPair{ - &dto.LabelPair{Name: proto.String("id"), Value: proto.String("2")}, + {Name: proto.String("id"), Value: proto.String("2")}, }, Value: proto.Float64(1.6), Timestamp: ts, }, nil, - &dto.Exemplar{ + { Label: []*dto.LabelPair{ - &dto.LabelPair{Name: proto.String("id"), Value: proto.String("3")}, + {Name: proto.String("id"), Value: proto.String("3")}, }, Value: proto.Float64(4), Timestamp: ts, }, - &dto.Exemplar{ + { Label: []*dto.LabelPair{ - &dto.LabelPair{Name: proto.String("id"), Value: proto.String("4")}, + {Name: proto.String("id"), Value: proto.String("4")}, }, Value: proto.Float64(4.5), Timestamp: ts, diff --git a/prometheus/metric.go b/prometheus/metric.go index 118a54e..48d4a5d 100644 --- a/prometheus/metric.go +++ b/prometheus/metric.go @@ -14,6 +14,8 @@ package prometheus import ( + "errors" + "sort" "strings" "time" @@ -158,3 +160,91 @@ func (m timestampedMetric) Write(pb *dto.Metric) error { func NewMetricWithTimestamp(t time.Time, m Metric) Metric { return timestampedMetric{Metric: m, t: t} } + +type withExemplarsMetric struct { + Metric + + exemplars []*dto.Exemplar +} + +func (m *withExemplarsMetric) Write(pb *dto.Metric) error { + if err := m.Metric.Write(pb); err != nil { + return err + } + + switch { + case pb.Counter != nil: + pb.Counter.Exemplar = m.exemplars[len(m.exemplars)-1] + case pb.Histogram != nil: + for _, e := range m.exemplars { + // pb.Histogram.Bucket are sorted by UpperBound. + i := sort.Search(len(pb.Histogram.Bucket), func(i int) bool { + return pb.Histogram.Bucket[i].GetUpperBound() >= e.GetValue() + }) + if i < len(pb.Histogram.Bucket) { + pb.Histogram.Bucket[i].Exemplar = e + } else { + // This is not possible as last bucket is Inf. + panic("no bucket was found for given exemplar value") + } + } + default: + // TODO(bwplotka): Implement Gauge? + return errors.New("cannot inject exemplar into Gauge, Summary or Untyped") + } + + return nil +} + +// Exemplar is easier to use, user-facing representation of *dto.Exemplar. +type Exemplar struct { + Value float64 + Labels Labels + // Optional. + // Default value (time.Time{}) indicates its empty, which should be + // understood as time.Now() time at the moment of creation of metric. + Timestamp time.Time +} + +// NewMetricWithExemplars returns a new Metric wrapping the provided Metric with given +// exemplars. Exemplars are validated. +// +// Only last applicable exemplar is injected from the list. +// For example for Counter it means last exemplar is injected. +// For Histogram, it means last applicable exemplar for each bucket is injected. +// +// NewMetricWithExemplars works best with MustNewConstMetric and +// MustNewConstHistogram, see example. +func NewMetricWithExemplars(m Metric, exemplars ...Exemplar) (Metric, error) { + if len(exemplars) == 0 { + return nil, errors.New("no exemplar was passed for NewMetricWithExemplars") + } + + var ( + now = time.Now() + exs = make([]*dto.Exemplar, len(exemplars)) + err error + ) + for i, e := range exemplars { + ts := e.Timestamp + if ts == (time.Time{}) { + ts = now + } + exs[i], err = newExemplar(e.Value, ts, e.Labels) + if err != nil { + return nil, err + } + } + + return &withExemplarsMetric{Metric: m, exemplars: exs}, nil +} + +// MustNewMetricWithExemplars is a version of NewMetricWithExemplars that panics where +// NewMetricWithExemplars would have returned an error. +func MustNewMetricWithExemplars(m Metric, exemplars ...Exemplar) Metric { + ret, err := NewMetricWithExemplars(m, exemplars...) + if err != nil { + panic(err) + } + return ret +} diff --git a/prometheus/metric_test.go b/prometheus/metric_test.go index 7145f5e..6100e0d 100644 --- a/prometheus/metric_test.go +++ b/prometheus/metric_test.go @@ -13,7 +13,12 @@ package prometheus -import "testing" +import ( + "testing" + + "github.com/golang/protobuf/proto" + dto "github.com/prometheus/client_model/go" +) func TestBuildFQName(t *testing.T) { scenarios := []struct{ namespace, subsystem, name, result string }{ @@ -33,3 +38,41 @@ func TestBuildFQName(t *testing.T) { } } } + +func TestWithExemplarsMetric(t *testing.T) { + t.Run("histogram", func(t *testing.T) { + // Create a constant histogram from values we got from a 3rd party telemetry system. + h := MustNewConstHistogram( + NewDesc("http_request_duration_seconds", "A histogram of the HTTP request durations.", nil, nil), + 4711, 403.34, + map[float64]uint64{25: 121, 50: 2403, 100: 3221, 200: 4233}, + ) + + m := &withExemplarsMetric{Metric: h, exemplars: []*dto.Exemplar{ + {Value: proto.Float64(24.0)}, + {Value: proto.Float64(25.1)}, + {Value: proto.Float64(42.0)}, + {Value: proto.Float64(89.0)}, + {Value: proto.Float64(100.0)}, + {Value: proto.Float64(157.0)}, + }} + metric := dto.Metric{} + if err := m.Write(&metric); err != nil { + t.Fatal(err) + } + if want, got := 4, len(metric.GetHistogram().Bucket); want != got { + t.Errorf("want %v, got %v", want, got) + } + + expectedExemplarVals := []float64{24.0, 42.0, 100.0, 157.0} + for i, b := range metric.GetHistogram().Bucket { + if b.Exemplar == nil { + t.Errorf("Expected exemplar for bucket %v, got nil", i) + } + if want, got := expectedExemplarVals[i], *metric.GetHistogram().Bucket[i].Exemplar.Value; want != got { + t.Errorf("%v: want %v, got %v", i, want, got) + } + } + }) + +} From 9894406186229c06d763d4bb6595d9260dcd9de2 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Thu, 17 Mar 2022 11:31:49 +0000 Subject: [PATCH 09/37] Fixed lint warning. Signed-off-by: Bartlomiej Plotka --- prometheus/metric_test.go | 1 + 1 file changed, 1 insertion(+) diff --git a/prometheus/metric_test.go b/prometheus/metric_test.go index 6100e0d..445b838 100644 --- a/prometheus/metric_test.go +++ b/prometheus/metric_test.go @@ -16,6 +16,7 @@ package prometheus import ( "testing" + //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility. "github.com/golang/protobuf/proto" dto "github.com/prometheus/client_model/go" ) From 5d78aaad41f50d2278c770fa9ac96fce76d8ec8d Mon Sep 17 00:00:00 2001 From: Kemal Akkoyun Date: Thu, 17 Mar 2022 16:34:12 +0100 Subject: [PATCH 10/37] .circleci: Add config to test against go1.18 (#1006) * Add config to test against go1.18 Signed-off-by: Kemal Akkoyun * Try to fix circleci Signed-off-by: Kemal Akkoyun --- .circleci/config.yml | 11 +++-- .github/workflows/golangci-lint.yml | 1 + prometheus/gen_go_collector_metrics_set.go | 10 ++++- prometheus/go_collector_metrics_go118_test.go | 41 +++++++++++++++++++ 4 files changed, 58 insertions(+), 5 deletions(-) create mode 100644 prometheus/go_collector_metrics_go118_test.go diff --git a/.circleci/config.yml b/.circleci/config.yml index 9bb9886..1525318 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -1,6 +1,6 @@ version: 2.1 orbs: - go: circleci/go@0.2.0 + go: circleci/go@1.7.1 prometheus: prometheus/prometheus@0.16.0 jobs: test: @@ -17,8 +17,7 @@ jobs: type: boolean default: true docker: - - image: circleci/golang:<< parameters.go_version >> - working_directory: /go/src/github.com/prometheus/client_golang + - image: cimg/go:<< parameters.go_version >> steps: - checkout - when: @@ -59,4 +58,10 @@ workflows: name: go-1-17 go_version: "1.17" run_lint: true + - test: + name: go-1-18 + go_version: "1.18" + run_lint: true + # Style and unused/missing packages are only checked against + # the latest supported Go version. run_style_and_unused: true diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index 8211b9a..a4622b9 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -20,6 +20,7 @@ jobs: - name: install Go uses: actions/setup-go@v2 with: + # golangci-lint is not read for Go 1.18 (https://github.com/golangci/golangci-lint/issues/2649) go-version: 1.17.x - name: Lint uses: golangci/golangci-lint-action@v3.1.0 diff --git a/prometheus/gen_go_collector_metrics_set.go b/prometheus/gen_go_collector_metrics_set.go index e33b397..4bc127c 100644 --- a/prometheus/gen_go_collector_metrics_set.go +++ b/prometheus/gen_go_collector_metrics_set.go @@ -38,8 +38,10 @@ func main() { log.Fatal("requires Go version (e.g. go1.17) as an argument") } toolVersion := runtime.Version() - if majorVersion := toolVersion[:strings.LastIndexByte(toolVersion, '.')]; majorVersion != os.Args[1] { - log.Fatalf("using Go version %q but expected Go version %q", majorVersion, os.Args[1]) + mtv := majorVersion(toolVersion) + mv != majorVersion(os.Args[1]) + if mtv != mv { + log.Fatalf("using Go version %q but expected Go version %q", mtv, mv) } version, err := parseVersion(os.Args[1]) if err != nil { @@ -93,6 +95,10 @@ func parseVersion(s string) (goVersion, error) { return goVersion(i), err } +func majorVersion(v string) string { + return v[:strings.LastIndexByte(v, '.')] +} + func rmCardinality() int { cardinality := 0 diff --git a/prometheus/go_collector_metrics_go118_test.go b/prometheus/go_collector_metrics_go118_test.go new file mode 100644 index 0000000..2bcf545 --- /dev/null +++ b/prometheus/go_collector_metrics_go118_test.go @@ -0,0 +1,41 @@ +// Code generated by gen_go_collector_metrics_set.go; DO NOT EDIT. +//go:generate go run gen_go_collector_metrics_set.go go1.18 + +//go:build go1.18 && !go1.19 +// +build go1.18,!go1.19 + +package prometheus + +var expectedRuntimeMetrics = map[string]string{ + "/gc/cycles/automatic:gc-cycles": "go_gc_cycles_automatic_gc_cycles_total", + "/gc/cycles/forced:gc-cycles": "go_gc_cycles_forced_gc_cycles_total", + "/gc/cycles/total:gc-cycles": "go_gc_cycles_total_gc_cycles_total", + "/gc/heap/allocs-by-size:bytes": "go_gc_heap_allocs_by_size_bytes_total", + "/gc/heap/allocs:bytes": "go_gc_heap_allocs_bytes_total", + "/gc/heap/allocs:objects": "go_gc_heap_allocs_objects_total", + "/gc/heap/frees-by-size:bytes": "go_gc_heap_frees_by_size_bytes_total", + "/gc/heap/frees:bytes": "go_gc_heap_frees_bytes_total", + "/gc/heap/frees:objects": "go_gc_heap_frees_objects_total", + "/gc/heap/goal:bytes": "go_gc_heap_goal_bytes", + "/gc/heap/objects:objects": "go_gc_heap_objects_objects", + "/gc/heap/tiny/allocs:objects": "go_gc_heap_tiny_allocs_objects_total", + "/gc/pauses:seconds": "go_gc_pauses_seconds_total", + "/memory/classes/heap/free:bytes": "go_memory_classes_heap_free_bytes", + "/memory/classes/heap/objects:bytes": "go_memory_classes_heap_objects_bytes", + "/memory/classes/heap/released:bytes": "go_memory_classes_heap_released_bytes", + "/memory/classes/heap/stacks:bytes": "go_memory_classes_heap_stacks_bytes", + "/memory/classes/heap/unused:bytes": "go_memory_classes_heap_unused_bytes", + "/memory/classes/metadata/mcache/free:bytes": "go_memory_classes_metadata_mcache_free_bytes", + "/memory/classes/metadata/mcache/inuse:bytes": "go_memory_classes_metadata_mcache_inuse_bytes", + "/memory/classes/metadata/mspan/free:bytes": "go_memory_classes_metadata_mspan_free_bytes", + "/memory/classes/metadata/mspan/inuse:bytes": "go_memory_classes_metadata_mspan_inuse_bytes", + "/memory/classes/metadata/other:bytes": "go_memory_classes_metadata_other_bytes", + "/memory/classes/os-stacks:bytes": "go_memory_classes_os_stacks_bytes", + "/memory/classes/other:bytes": "go_memory_classes_other_bytes", + "/memory/classes/profiling/buckets:bytes": "go_memory_classes_profiling_buckets_bytes", + "/memory/classes/total:bytes": "go_memory_classes_total_bytes", + "/sched/goroutines:goroutines": "go_sched_goroutines_goroutines", + "/sched/latencies:seconds": "go_sched_latencies_seconds", +} + +const expectedRuntimeMetricsCardinality = 79 From 36b47eb0ab8eec466411a5e9f7d5156d988b757f Mon Sep 17 00:00:00 2001 From: zhijian Date: Thu, 17 Mar 2022 23:43:50 +0800 Subject: [PATCH 11/37] When prefix is empty, no more dots should be written (#1005) Signed-off-by: zhijian --- prometheus/graphite/bridge.go | 12 +++++++----- prometheus/graphite/bridge_test.go | 14 ++++++++++---- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/prometheus/graphite/bridge.go b/prometheus/graphite/bridge.go index 09b7c9e..09c31b5 100644 --- a/prometheus/graphite/bridge.go +++ b/prometheus/graphite/bridge.go @@ -197,14 +197,16 @@ func writeMetrics(w io.Writer, mfs []*dto.MetricFamily, useTags bool, prefix str buf := bufio.NewWriter(w) for _, s := range vec { - for _, c := range prefix { - if _, err := buf.WriteRune(c); err != nil { + if prefix != "" { + for _, c := range prefix { + if _, err := buf.WriteRune(c); err != nil { + return err + } + } + if err := buf.WriteByte('.'); err != nil { return err } } - if err := buf.WriteByte('.'); err != nil { - return err - } if err := writeMetric(buf, s.Metric, useTags); err != nil { return err } diff --git a/prometheus/graphite/bridge_test.go b/prometheus/graphite/bridge_test.go index ae418af..df0cfff 100644 --- a/prometheus/graphite/bridge_test.go +++ b/prometheus/graphite/bridge_test.go @@ -101,6 +101,7 @@ func testWriteSummary(t *testing.T, useTags bool) { {prefix: "prefix"}, {prefix: "pre/fix"}, {prefix: "pre.fix"}, + {prefix: ""}, } var ( @@ -141,10 +142,15 @@ func testWriteSummary(t *testing.T, useTags bool) { t.Fatalf("error: %v", err) } - wantWithPrefix := fmt.Sprintf(want, - tc.prefix, tc.prefix, tc.prefix, tc.prefix, tc.prefix, - tc.prefix, tc.prefix, tc.prefix, tc.prefix, tc.prefix, - ) + var wantWithPrefix string + if tc.prefix == "" { + wantWithPrefix = strings.ReplaceAll(want, "%s.", "") + } else { + wantWithPrefix = fmt.Sprintf(want, + tc.prefix, tc.prefix, tc.prefix, tc.prefix, tc.prefix, + tc.prefix, tc.prefix, tc.prefix, tc.prefix, tc.prefix, + ) + } got := buf.String() From 6559749c87b4b9917e8c5f15049b6f4e3bfb3e94 Mon Sep 17 00:00:00 2001 From: Kemal Akkoyun Date: Thu, 17 Mar 2022 16:44:57 +0100 Subject: [PATCH 12/37] Add statebot config (#1001) * Add statebot config Signed-off-by: Kemal Akkoyun * Disable for milestones Remove references to remind bot Signed-off-by: Kemal Akkoyun --- .github/stale.yml | 88 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 .github/stale.yml diff --git a/.github/stale.yml b/.github/stale.yml new file mode 100644 index 0000000..d9fa51d --- /dev/null +++ b/.github/stale.yml @@ -0,0 +1,88 @@ +# Configuration for probot-stale - https://github.com/probot/stale + +# Number of days of inactivity before an Issue or Pull Request becomes stale +# daysUntilStale: 30 + +# Number of days of inactivity before an Issue or Pull Request with the stale label is closed. +# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale. +# daysUntilClose: 7 + +# NOTICE: Check below for the individual settings for each type of event. + +# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled) +onlyLabels: [] + +# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable +exemptLabels: + - pinned + - security + - keep-open + - wip + - "[Status] Maybe Later" + +# Set to true to ignore issues in a project (defaults to false) +exemptProjects: true # default: false + +# Set to true to ignore issues in a milestone (defaults to false) +exemptMilestones: true # default: false + +# Set to true to ignore issues with an assignee (defaults to false) +exemptAssignees: false + +# Label to use when marking as stale +staleLabel: stale + +# Comment to post when marking as stale. Set to `false` to disable +# markComment: > +# This issue has been automatically marked as stale because it has not had +# recent activity. It will be closed if no further activity occurs. Thank you +# for your contributions. + +# Comment to post when removing the stale label. +# unmarkComment: > +# Your comment here. + +# Comment to post when closing a stale Issue or Pull Request. +# closeComment: > +# Your comment here. + +# Limit the number of actions per hour, from 1-30. Default is 30 +limitPerRun: 30 + +# Limit to only `issues` or `pulls` +# only: issues + +# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls': +# pulls: +# daysUntilStale: 30 +# markComment: > +# This pull request has been automatically marked as stale because it has not had +# recent activity. It will be closed if no further activity occurs. Thank you +# for your contributions. + +# issues: +# exemptLabels: +# - confirmed + +pull: + daysUntilClose: 14 + daysUntilStale: 60 + markComment: > + Hello 👋 Looks like there was no activity on this amazing PR for the last 60 days. + **Do you mind updating us on the status?** Is there anything we can help with? If you plan to still work on it, just comment on this PR or push a commit. Thanks! 🤗 + + If there will be no activity in the next 2 weeks, this issue will be closed (we can always reopen a PR if you get back to this!). + # unmarkComment: No need for unmark comment. + closeComment: > + Closing for now as promised, let us know if you need this to be reopened! 🤗 +issues: + daysUntilClose: 90 + daysUntilStale: 180 + markComment: > + Hello 👋 Looks like there was no activity on this issue for the last 3 months. + **Do you mind updating us on the status?** Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 + + If there will be no activity in the next 4 weeks, this issue will be closed (we can always reopen an issue if we need!). + # unmarkComment: No need for unmark comment. + closeComment: > + Closing for now as promised, let us know if you need this to be reopened! 🤗 From 3bc8f2c651ff1abe956f970ae3d0a2d45737c501 Mon Sep 17 00:00:00 2001 From: PrometheusBot Date: Fri, 18 Mar 2022 12:00:13 +0100 Subject: [PATCH 13/37] Update common Prometheus files (#1009) Signed-off-by: prombot --- .github/workflows/golangci-lint.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index a4622b9..8211b9a 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -20,7 +20,6 @@ jobs: - name: install Go uses: actions/setup-go@v2 with: - # golangci-lint is not read for Go 1.18 (https://github.com/golangci/golangci-lint/issues/2649) go-version: 1.17.x - name: Lint uses: golangci/golangci-lint-action@v3.1.0 From 8dfa334295e85f9b1e48ce862fae5f337faa6d2f Mon Sep 17 00:00:00 2001 From: Mitsuo Heijo <25817501+johejo@users.noreply.github.com> Date: Mon, 28 Mar 2022 03:11:56 +0900 Subject: [PATCH 14/37] Remove workaround for pre go1.15 (#1010) Signed-off-by: Mitsuo Heijo --- prometheus/collectors/dbstats_collector.go | 4 +-- .../collectors/dbstats_collector_go115.go | 31 ------------------- .../collectors/dbstats_collector_pre_go115.go | 27 ---------------- 3 files changed, 2 insertions(+), 60 deletions(-) delete mode 100644 prometheus/collectors/dbstats_collector_go115.go delete mode 100644 prometheus/collectors/dbstats_collector_pre_go115.go diff --git a/prometheus/collectors/dbstats_collector.go b/prometheus/collectors/dbstats_collector.go index e09f149..d5a7279 100644 --- a/prometheus/collectors/dbstats_collector.go +++ b/prometheus/collectors/dbstats_collector.go @@ -101,7 +101,7 @@ func (c *dbStatsCollector) Describe(ch chan<- *prometheus.Desc) { ch <- c.waitDuration ch <- c.maxIdleClosed ch <- c.maxLifetimeClosed - c.describeNewInGo115(ch) + ch <- c.maxIdleTimeClosed } // Collect implements Collector. @@ -115,5 +115,5 @@ func (c *dbStatsCollector) Collect(ch chan<- prometheus.Metric) { ch <- prometheus.MustNewConstMetric(c.waitDuration, prometheus.CounterValue, stats.WaitDuration.Seconds()) ch <- prometheus.MustNewConstMetric(c.maxIdleClosed, prometheus.CounterValue, float64(stats.MaxIdleClosed)) ch <- prometheus.MustNewConstMetric(c.maxLifetimeClosed, prometheus.CounterValue, float64(stats.MaxLifetimeClosed)) - c.collectNewInGo115(ch, stats) + ch <- prometheus.MustNewConstMetric(c.maxIdleTimeClosed, prometheus.CounterValue, float64(stats.MaxIdleTimeClosed)) } diff --git a/prometheus/collectors/dbstats_collector_go115.go b/prometheus/collectors/dbstats_collector_go115.go deleted file mode 100644 index 6d152fb..0000000 --- a/prometheus/collectors/dbstats_collector_go115.go +++ /dev/null @@ -1,31 +0,0 @@ -// Copyright 2021 The Prometheus Authors -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//go:build go1.15 -// +build go1.15 - -package collectors - -import ( - "database/sql" - - "github.com/prometheus/client_golang/prometheus" -) - -func (c *dbStatsCollector) describeNewInGo115(ch chan<- *prometheus.Desc) { - ch <- c.maxIdleTimeClosed -} - -func (c *dbStatsCollector) collectNewInGo115(ch chan<- prometheus.Metric, stats sql.DBStats) { - ch <- prometheus.MustNewConstMetric(c.maxIdleTimeClosed, prometheus.CounterValue, float64(stats.MaxIdleTimeClosed)) -} diff --git a/prometheus/collectors/dbstats_collector_pre_go115.go b/prometheus/collectors/dbstats_collector_pre_go115.go deleted file mode 100644 index 6523506..0000000 --- a/prometheus/collectors/dbstats_collector_pre_go115.go +++ /dev/null @@ -1,27 +0,0 @@ -// Copyright 2021 The Prometheus Authors -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//go:build !go1.15 -// +build !go1.15 - -package collectors - -import ( - "database/sql" - - "github.com/prometheus/client_golang/prometheus" -) - -func (c *dbStatsCollector) describeNewInGo115(ch chan<- *prometheus.Desc) {} - -func (c *dbStatsCollector) collectNewInGo115(ch chan<- prometheus.Metric, stats sql.DBStats) {} From 29e8191aff2174cb56888658b4afd6a7940a9b3e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuel=20R=C3=BCger?= Date: Tue, 29 Mar 2022 15:15:42 +0200 Subject: [PATCH 15/37] go.mod: Update dependencies (#1018) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Manuel Rüger --- go.mod | 6 +++--- go.sum | 25 +++++++++++++++++++------ 2 files changed, 22 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 9088d08..d0ace2d 100644 --- a/go.mod +++ b/go.mod @@ -6,10 +6,10 @@ require ( github.com/golang/protobuf v1.5.2 github.com/json-iterator/go v1.1.12 github.com/prometheus/client_model v0.2.0 - github.com/prometheus/common v0.32.1 + github.com/prometheus/common v0.33.0 github.com/prometheus/procfs v0.7.3 - golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 - google.golang.org/protobuf v1.26.0 + golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886 + google.golang.org/protobuf v1.28.0 ) go 1.15 diff --git a/go.sum b/go.sum index 7aae554..df0996e 100644 --- a/go.sum +++ b/go.sum @@ -64,9 +64,11 @@ github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY= +github.com/go-kit/log v0.2.0/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0= github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= +github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= @@ -166,6 +168,7 @@ github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXP github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= +github.com/prometheus/client_golang v1.12.1/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= @@ -174,8 +177,9 @@ github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6T github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc= -github.com/prometheus/common v0.32.1 h1:hWIdL3N2HoUx3B8j3YN9mWor0qhY/NlEKZEaXxuIRh4= github.com/prometheus/common v0.32.1/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls= +github.com/prometheus/common v0.33.0 h1:rHgav/0a6+uYgGdNt3jwz8FNSesO/Hsang3O0T9A5SE= +github.com/prometheus/common v0.33.0/go.mod h1:gB3sOl7P0TvJabZpLY5uQMpUqRCPPCyRLCZYc7JZTNE= github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= @@ -264,15 +268,18 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= -golang.org/x/net v0.0.0-20210525063256-abc453219eb5 h1:wjuX4b5yYQnEQHzd+CBcrcC6OVR2J1CN6mUy0oSxIPo= golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc= +golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c h1:pkQiBZBvdos9qq4wBAHqlzuZHEXo07pqV06ef90u1WI= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= +golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b h1:clP8eMhB30EHdc0bd2Twtq6kgU7yl5ub2cQLSdrv1Dg= +golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -317,16 +324,21 @@ golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 h1:XfKQ4OlFl8okEOr5UvAqFRVj8pY/4yfcXrddB8qAbU0= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886 h1:eJv7u3ksNXoLbGSKuv2s/SIO4tJVxc/A+MTpzxDgz/Q= +golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= +golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -450,8 +462,9 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= -google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw= +google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= From e2504f86bb7cc57e05ce7089ff9a208bcfe23ad6 Mon Sep 17 00:00:00 2001 From: prombot Date: Thu, 31 Mar 2022 19:50:35 +0000 Subject: [PATCH 16/37] Update common Prometheus files Signed-off-by: prombot --- .github/workflows/golangci-lint.yml | 4 ++-- Makefile.common | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index 8211b9a..662ea3b 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -20,8 +20,8 @@ jobs: - name: install Go uses: actions/setup-go@v2 with: - go-version: 1.17.x + go-version: 1.18.x - name: Lint uses: golangci/golangci-lint-action@v3.1.0 with: - version: v1.44.2 + version: v1.45.2 diff --git a/Makefile.common b/Makefile.common index 5ab1b14..c263b73 100644 --- a/Makefile.common +++ b/Makefile.common @@ -83,7 +83,7 @@ PROMU_URL := https://github.com/prometheus/promu/releases/download/v$(PROMU_ GOLANGCI_LINT := GOLANGCI_LINT_OPTS ?= -GOLANGCI_LINT_VERSION ?= v1.44.2 +GOLANGCI_LINT_VERSION ?= v1.45.2 # golangci-lint only supports linux, darwin and windows platforms on i386/amd64. # windows isn't included here because of the path separator being different. ifeq ($(GOHOSTOS),$(filter $(GOHOSTOS),linux darwin)) From 40e54a75a604a3d15ce928f1a21c2a47bfee0d8d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1=C5=A1=20Dohn=C3=A1lek?= Date: Thu, 7 Apr 2022 09:13:14 +0200 Subject: [PATCH 17/37] Refactor apiClientImpl.DoGetFallback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make apiClientImpl.DoGetFallback more idiomatic and efficient: * Save result of args.Encode() operation as it might be used 2 times in the function and due to looping and sorting it might be heavy. * Follow line-of-sight practise and therefore simplify the code. Signed-off-by: Tomáš Dohnálek --- api/prometheus/v1/api.go | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/api/prometheus/v1/api.go b/api/prometheus/v1/api.go index 857512f..0be68fa 100644 --- a/api/prometheus/v1/api.go +++ b/api/prometheus/v1/api.go @@ -1133,7 +1133,8 @@ func (h *apiClientImpl) Do(ctx context.Context, req *http.Request) (*http.Respon // DoGetFallback will attempt to do the request as-is, and on a 405 or 501 it // will fallback to a GET request. func (h *apiClientImpl) DoGetFallback(ctx context.Context, u *url.URL, args url.Values) (*http.Response, []byte, Warnings, error) { - req, err := http.NewRequest(http.MethodPost, u.String(), strings.NewReader(args.Encode())) + encodedArgs := args.Encode() + req, err := http.NewRequest(http.MethodPost, u.String(), strings.NewReader(encodedArgs)) if err != nil { return nil, nil, nil, err } @@ -1141,19 +1142,14 @@ func (h *apiClientImpl) DoGetFallback(ctx context.Context, u *url.URL, args url. resp, body, warnings, err := h.Do(ctx, req) if resp != nil && (resp.StatusCode == http.StatusMethodNotAllowed || resp.StatusCode == http.StatusNotImplemented) { - u.RawQuery = args.Encode() + u.RawQuery = encodedArgs req, err = http.NewRequest(http.MethodGet, u.String(), nil) if err != nil { return nil, nil, warnings, err } - - } else { - if err != nil { - return resp, body, warnings, err - } - return resp, body, warnings, nil + return h.Do(ctx, req) } - return h.Do(ctx, req) + return resp, body, warnings, err } func formatTime(t time.Time) string { From 06b641214cbf9be14c0011ae6a60a0831d07fb6c Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Tue, 12 Apr 2022 16:24:39 +0200 Subject: [PATCH 18/37] Added info about our slack channel. (#1029) --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 6b0e874..0d09ba0 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,7 @@ [![CircleCI](https://circleci.com/gh/prometheus/client_golang/tree/master.svg?style=svg)](https://circleci.com/gh/prometheus/client_golang/tree/master) [![Go Report Card](https://goreportcard.com/badge/github.com/prometheus/client_golang)](https://goreportcard.com/report/github.com/prometheus/client_golang) [![Go Reference](https://pkg.go.dev/badge/github.com/prometheus/client_golang.svg)](https://pkg.go.dev/github.com/prometheus/client_golang) +[![Slack](https://img.shields.io/badge/join%20slack-%23prometheus--client_golang-brightgreen.svg)](https://slack.cncf.io/) This is the [Go](http://golang.org) client library for [Prometheus](http://prometheus.io). It has two separate parts, one for @@ -66,3 +67,5 @@ The `extraction` and `text` packages are now contained in See the [contributing guidelines](CONTRIBUTING.md) and the [Community section](http://prometheus.io/community/) of the homepage. + +`clint_golang` community is also present on the CNCF Slack `#prometheus-client_golang`. From 0bab4fda94f40bc0147c43dfc46a3308bc5c838d Mon Sep 17 00:00:00 2001 From: Tatsuhiro Tsujikawa <404610+tatsuhiro-t@users.noreply.github.com> Date: Tue, 12 Apr 2022 23:25:43 +0900 Subject: [PATCH 19/37] push: Add PushContext and AddContext to Pusher (#1028) Add PushContext and AddContext to Pusher, which are context-aware version of Push and Add respectively. They give a caller the ability to cancel an HTTP request. Signed-off-by: Tatsuhiro Tsujikawa --- prometheus/push/push.go | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/prometheus/push/push.go b/prometheus/push/push.go index c1a6cb9..3bb1466 100644 --- a/prometheus/push/push.go +++ b/prometheus/push/push.go @@ -36,6 +36,7 @@ package push import ( "bytes" + "context" "encoding/base64" "errors" "fmt" @@ -123,14 +124,28 @@ func New(url, job string) *Pusher { // Push returns the first error encountered by any method call (including this // one) in the lifetime of the Pusher. func (p *Pusher) Push() error { - return p.push(http.MethodPut) + return p.push(context.Background(), http.MethodPut) +} + +// PushContext is like Push but includes a context. +// +// If the context expires before HTTP request is complete, an error is returned. +func (p *Pusher) PushContext(ctx context.Context) error { + return p.push(ctx, http.MethodPut) } // Add works like push, but only previously pushed metrics with the same name // (and the same job and other grouping labels) will be replaced. (It uses HTTP // method “POST” to push to the Pushgateway.) func (p *Pusher) Add() error { - return p.push(http.MethodPost) + return p.push(context.Background(), http.MethodPost) +} + +// AddContext is like Add but includes a context. +// +// If the context expires before HTTP request is complete, an error is returned. +func (p *Pusher) AddContext(ctx context.Context) error { + return p.push(ctx, http.MethodPost) } // Gatherer adds a Gatherer to the Pusher, from which metrics will be gathered @@ -233,7 +248,7 @@ func (p *Pusher) Delete() error { return nil } -func (p *Pusher) push(method string) error { +func (p *Pusher) push(ctx context.Context, method string) error { if p.error != nil { return p.error } @@ -260,7 +275,7 @@ func (p *Pusher) push(method string) error { } enc.Encode(mf) } - req, err := http.NewRequest(method, p.fullURL(), buf) + req, err := http.NewRequestWithContext(ctx, method, p.fullURL(), buf) if err != nil { return err } From cc7991d97715ab6ad19da54476a0c633e7bc1880 Mon Sep 17 00:00:00 2001 From: Tomas Dohnalek Date: Wed, 13 Apr 2022 07:21:50 +0200 Subject: [PATCH 20/37] Make Query requests idempotent (#1022) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Make Query requests idempotent Address #1020. Signed-off-by: Tomáš Dohnálek * Use empty header Signed-off-by: Tomáš Dohnálek * Document issue with original documentation Signed-off-by: Tomáš Dohnálek --- api/prometheus/v1/api.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/api/prometheus/v1/api.go b/api/prometheus/v1/api.go index 0be68fa..5ed0912 100644 --- a/api/prometheus/v1/api.go +++ b/api/prometheus/v1/api.go @@ -1139,6 +1139,14 @@ func (h *apiClientImpl) DoGetFallback(ctx context.Context, u *url.URL, args url. return nil, nil, nil, err } req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + // Following comment originates from https://pkg.go.dev/net/http#Transport + // Transport only retries a request upon encountering a network error if the request is + // idempotent and either has no body or has its Request.GetBody defined. HTTP requests + // are considered idempotent if they have HTTP methods GET, HEAD, OPTIONS, or TRACE; or + // if their Header map contains an "Idempotency-Key" or "X-Idempotency-Key" entry. If the + // idempotency key value is a zero-length slice, the request is treated as idempotent but + // the header is not sent on the wire. + req.Header["Idempotency-Key"] = nil resp, body, warnings, err := h.Do(ctx, req) if resp != nil && (resp.StatusCode == http.StatusMethodNotAllowed || resp.StatusCode == http.StatusNotImplemented) { From 24172847e35ba46025c49d90b8846b59eb5d9ead Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 13 Apr 2022 10:55:22 +0200 Subject: [PATCH 21/37] gocollector: Added options to Go Collector for changing the (#1031) * Renamed files. Signed-off-by: Bartlomiej Plotka * gocollector: Added options to Go Collector for diffetent collections. Fixes https://github.com/prometheus/client_golang/issues/983 Also: * fixed TestMemStatsEquivalence, it was noop before (: * Removed gc_cpu_fraction metric completely, since it's not working completely for Go1.17+ Signed-off-by: Bartlomiej Plotka --- prometheus/collectors/collectors.go | 24 ++ ...{go_collector.go => go_collector_go116.go} | 26 +- prometheus/collectors/go_collector_latest.go | 91 +++++++ prometheus/go_collector.go | 10 +- prometheus/go_collector_go116.go | 17 +- ...lector_go117.go => go_collector_latest.go} | 223 +++++++++++++----- ...17_test.go => go_collector_latest_test.go} | 168 +++++++------ 7 files changed, 398 insertions(+), 161 deletions(-) rename prometheus/collectors/{go_collector.go => go_collector_go116.go} (64%) create mode 100644 prometheus/collectors/go_collector_latest.go rename prometheus/{go_collector_go117.go => go_collector_latest.go} (61%) rename prometheus/{go_collector_go117_test.go => go_collector_latest_test.go} (72%) diff --git a/prometheus/collectors/collectors.go b/prometheus/collectors/collectors.go index c4d0f5c..f4c9291 100644 --- a/prometheus/collectors/collectors.go +++ b/prometheus/collectors/collectors.go @@ -14,3 +14,27 @@ // Package collectors provides implementations of prometheus.Collector to // conveniently collect process and Go-related metrics. package collectors + +import "github.com/prometheus/client_golang/prometheus" + +// NewBuildInfoCollector returns a collector collecting a single metric +// "go_build_info" with the constant value 1 and three labels "path", "version", +// and "checksum". Their label values contain the main module path, version, and +// checksum, respectively. The labels will only have meaningful values if the +// binary is built with Go module support and from source code retrieved from +// the source repository (rather than the local file system). This is usually +// accomplished by building from outside of GOPATH, specifying the full address +// of the main package, e.g. "GO111MODULE=on go run +// github.com/prometheus/client_golang/examples/random". If built without Go +// module support, all label values will be "unknown". If built with Go module +// support but using the source code from the local file system, the "path" will +// be set appropriately, but "checksum" will be empty and "version" will be +// "(devel)". +// +// This collector uses only the build information for the main module. See +// https://github.com/povilasv/prommod for an example of a collector for the +// module dependencies. +func NewBuildInfoCollector() prometheus.Collector { + //nolint:staticcheck // Ignore SA1019 until v2. + return prometheus.NewBuildInfoCollector() +} diff --git a/prometheus/collectors/go_collector.go b/prometheus/collectors/go_collector_go116.go similarity index 64% rename from prometheus/collectors/go_collector.go rename to prometheus/collectors/go_collector_go116.go index edaa4e5..effc578 100644 --- a/prometheus/collectors/go_collector.go +++ b/prometheus/collectors/go_collector_go116.go @@ -11,6 +11,9 @@ // See the License for the specific language governing permissions and // limitations under the License. +//go:build !go1.17 +// +build !go1.17 + package collectors import "github.com/prometheus/client_golang/prometheus" @@ -42,28 +45,5 @@ import "github.com/prometheus/client_golang/prometheus" // NOTE: The problem is solved in Go 1.15, see // https://github.com/golang/go/issues/19812 for the related Go issue. func NewGoCollector() prometheus.Collector { - //nolint:staticcheck // Ignore SA1019 until v2. return prometheus.NewGoCollector() } - -// NewBuildInfoCollector returns a collector collecting a single metric -// "go_build_info" with the constant value 1 and three labels "path", "version", -// and "checksum". Their label values contain the main module path, version, and -// checksum, respectively. The labels will only have meaningful values if the -// binary is built with Go module support and from source code retrieved from -// the source repository (rather than the local file system). This is usually -// accomplished by building from outside of GOPATH, specifying the full address -// of the main package, e.g. "GO111MODULE=on go run -// github.com/prometheus/client_golang/examples/random". If built without Go -// module support, all label values will be "unknown". If built with Go module -// support but using the source code from the local file system, the "path" will -// be set appropriately, but "checksum" will be empty and "version" will be -// "(devel)". -// -// This collector uses only the build information for the main module. See -// https://github.com/povilasv/prommod for an example of a collector for the -// module dependencies. -func NewBuildInfoCollector() prometheus.Collector { - //nolint:staticcheck // Ignore SA1019 until v2. - return prometheus.NewBuildInfoCollector() -} diff --git a/prometheus/collectors/go_collector_latest.go b/prometheus/collectors/go_collector_latest.go new file mode 100644 index 0000000..a4657a4 --- /dev/null +++ b/prometheus/collectors/go_collector_latest.go @@ -0,0 +1,91 @@ +// Copyright 2021 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build go1.17 +// +build go1.17 + +package collectors + +import "github.com/prometheus/client_golang/prometheus" + +//nolint:staticcheck // Ignore SA1019 until v2. +type goOptions = prometheus.GoCollectorOptions +type goOption func(o *goOptions) + +type GoCollectionOption uint32 + +const ( + // GoRuntimeMemStatsCollection represents the metrics represented by runtime.MemStats structure such as + // go_memstats_alloc_bytes + // go_memstats_alloc_bytes_total + // go_memstats_sys_bytes + // go_memstats_lookups_total + // go_memstats_mallocs_total + // go_memstats_frees_total + // go_memstats_heap_alloc_bytes + // go_memstats_heap_sys_bytes + // go_memstats_heap_idle_bytes + // go_memstats_heap_inuse_bytes + // go_memstats_heap_released_bytes + // go_memstats_heap_objects + // go_memstats_stack_inuse_bytes + // go_memstats_stack_sys_bytes + // go_memstats_mspan_inuse_bytes + // go_memstats_mspan_sys_bytes + // go_memstats_mcache_inuse_bytes + // go_memstats_mcache_sys_bytes + // go_memstats_buck_hash_sys_bytes + // go_memstats_gc_sys_bytes + // go_memstats_other_sys_bytes + // go_memstats_next_gc_bytes + // so the metrics known from pre client_golang v1.12.0, except skipped go_memstats_gc_cpu_fraction (see + // https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034 for explanation. + // + // NOTE that this mode represents runtime.MemStats statistics, but they are + // actually implemented using new runtime/metrics package. + // Deprecated: Use GoRuntimeMetricsCollection instead going forward. + GoRuntimeMemStatsCollection GoCollectionOption = 1 << iota + // GoRuntimeMetricsCollection is the new set of metrics represented by runtime/metrics package and follows + // consistent naming. The exposed metric set depends on Go version, but it is controlled against + // unexpected cardinality. This set has overlapping information with GoRuntimeMemStatsCollection, just with + // new names. GoRuntimeMetricsCollection is what is recommended for using going forward. + GoRuntimeMetricsCollection +) + +// WithGoCollections allows enabling different collections for Go collector on top of base metrics +// like go_goroutines, go_threads, go_gc_duration_seconds, go_memstats_last_gc_time_seconds, go_info. +// +// Check GoRuntimeMemStatsCollection and GoRuntimeMetricsCollection for more details. You can use none, +// one or more collections at once. For example: +// WithGoCollections(GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection) means both GoRuntimeMemStatsCollection +// metrics and GoRuntimeMetricsCollection will be exposed. +// +// Use WithGoCollections(GoRuntimeMemStatsCollection) to have Go collector working in +// the compatibility mode with client_golang pre v1.12 (move to runtime/metrics). +func WithGoCollections(flags uint32) goOption { + return func(o *goOptions) { + o.EnabledCollections = flags + } +} + +// NewGoCollector returns a collector that exports metrics about the current Go +// process using debug.GCStats using runtime/metrics. +func NewGoCollector(opts ...goOption) prometheus.Collector { + //nolint:staticcheck // Ignore SA1019 until v2. + promPkgOpts := make([]func(o *prometheus.GoCollectorOptions), len(opts)) + for i, opt := range opts { + promPkgOpts[i] = opt + } + //nolint:staticcheck // Ignore SA1019 until v2. + return prometheus.NewGoCollector(promPkgOpts...) +} diff --git a/prometheus/go_collector.go b/prometheus/go_collector.go index 08195b4..4d792aa 100644 --- a/prometheus/go_collector.go +++ b/prometheus/go_collector.go @@ -197,14 +197,6 @@ func goRuntimeMemStats() memStatsMetrics { ), eval: func(ms *runtime.MemStats) float64 { return float64(ms.NextGC) }, valType: GaugeValue, - }, { - desc: NewDesc( - memstatNamespace("gc_cpu_fraction"), - "The fraction of this program's available CPU time used by the GC since the program started.", - nil, nil, - ), - eval: func(ms *runtime.MemStats) float64 { return ms.GCCPUFraction }, - valType: GaugeValue, }, } } @@ -268,7 +260,6 @@ func (c *baseGoCollector) Collect(ch chan<- Metric) { quantiles[0.0] = stats.PauseQuantiles[0].Seconds() ch <- MustNewConstSummary(c.gcDesc, uint64(stats.NumGC), stats.PauseTotal.Seconds(), quantiles) ch <- MustNewConstMetric(c.gcLastTimeDesc, GaugeValue, float64(stats.LastGC.UnixNano())/1e9) - ch <- MustNewConstMetric(c.goInfoDesc, GaugeValue, 1) } @@ -278,6 +269,7 @@ func memstatNamespace(s string) string { // memStatsMetrics provide description, evaluator, runtime/metrics name, and // value type for memstat metrics. +// TODO(bwplotka): Remove with end Go 1.16 EOL and replace with runtime/metrics.Description type memStatsMetrics []struct { desc *Desc eval func(*runtime.MemStats) float64 diff --git a/prometheus/go_collector_go116.go b/prometheus/go_collector_go116.go index 2452613..897a6e9 100644 --- a/prometheus/go_collector_go116.go +++ b/prometheus/go_collector_go116.go @@ -40,13 +40,28 @@ type goCollector struct { // // Deprecated: Use collectors.NewGoCollector instead. func NewGoCollector() Collector { + msMetrics := goRuntimeMemStats() + msMetrics = append(msMetrics, struct { + desc *Desc + eval func(*runtime.MemStats) float64 + valType ValueType + }{ + // This metric is omitted in Go1.17+, see https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034 + desc: NewDesc( + memstatNamespace("gc_cpu_fraction"), + "The fraction of this program's available CPU time used by the GC since the program started.", + nil, nil, + ), + eval: func(ms *runtime.MemStats) float64 { return ms.GCCPUFraction }, + valType: GaugeValue, + }) return &goCollector{ base: newBaseGoCollector(), msLast: &runtime.MemStats{}, msRead: runtime.ReadMemStats, msMaxWait: time.Second, msMaxAge: 5 * time.Minute, - msMetrics: goRuntimeMemStats(), + msMetrics: msMetrics, } } diff --git a/prometheus/go_collector_go117.go b/prometheus/go_collector_latest.go similarity index 61% rename from prometheus/go_collector_go117.go rename to prometheus/go_collector_latest.go index d43bdcd..944794f 100644 --- a/prometheus/go_collector_go117.go +++ b/prometheus/go_collector_latest.go @@ -29,7 +29,66 @@ import ( dto "github.com/prometheus/client_model/go" ) +const ( + goGCHeapTinyAllocsObjects = "/gc/heap/tiny/allocs:objects" + goGCHeapAllocsObjects = "/gc/heap/allocs:objects" + goGCHeapFreesObjects = "/gc/heap/frees:objects" + goGCHeapAllocsBytes = "/gc/heap/allocs:bytes" + goGCHeapObjects = "/gc/heap/objects:objects" + goGCHeapGoalBytes = "/gc/heap/goal:bytes" + goMemoryClassesTotalBytes = "/memory/classes/total:bytes" + goMemoryClassesHeapObjectsBytes = "/memory/classes/heap/objects:bytes" + goMemoryClassesHeapUnusedBytes = "/memory/classes/heap/unused:bytes" + goMemoryClassesHeapReleasedBytes = "/memory/classes/heap/released:bytes" + goMemoryClassesHeapFreeBytes = "/memory/classes/heap/free:bytes" + goMemoryClassesHeapStacksBytes = "/memory/classes/heap/stacks:bytes" + goMemoryClassesOSStacksBytes = "/memory/classes/os-stacks:bytes" + goMemoryClassesMetadataMSpanInuseBytes = "/memory/classes/metadata/mspan/inuse:bytes" + goMemoryClassesMetadataMSPanFreeBytes = "/memory/classes/metadata/mspan/free:bytes" + goMemoryClassesMetadataMCacheInuseBytes = "/memory/classes/metadata/mcache/inuse:bytes" + goMemoryClassesMetadataMCacheFreeBytes = "/memory/classes/metadata/mcache/free:bytes" + goMemoryClassesProfilingBucketsBytes = "/memory/classes/profiling/buckets:bytes" + goMemoryClassesMetadataOtherBytes = "/memory/classes/metadata/other:bytes" + goMemoryClassesOtherBytes = "/memory/classes/other:bytes" +) + +// runtime/metrics names required for runtimeMemStats like logic. +var rmForMemStats = []string{goGCHeapTinyAllocsObjects, + goGCHeapAllocsObjects, + goGCHeapFreesObjects, + goGCHeapAllocsBytes, + goGCHeapObjects, + goGCHeapGoalBytes, + goMemoryClassesTotalBytes, + goMemoryClassesHeapObjectsBytes, + goMemoryClassesHeapUnusedBytes, + goMemoryClassesHeapReleasedBytes, + goMemoryClassesHeapFreeBytes, + goMemoryClassesHeapStacksBytes, + goMemoryClassesOSStacksBytes, + goMemoryClassesMetadataMSpanInuseBytes, + goMemoryClassesMetadataMSPanFreeBytes, + goMemoryClassesMetadataMCacheInuseBytes, + goMemoryClassesMetadataMCacheFreeBytes, + goMemoryClassesProfilingBucketsBytes, + goMemoryClassesMetadataOtherBytes, + goMemoryClassesOtherBytes, +} + +func bestEffortLookupRM(lookup []string) []metrics.Description { + ret := make([]metrics.Description, 0, len(lookup)) + for _, rm := range metrics.All() { + for _, m := range lookup { + if m == rm.Name { + ret = append(ret, rm) + } + } + } + return ret +} + type goCollector struct { + opt GoCollectorOptions base baseGoCollector // mu protects updates to all fields ensuring a consistent @@ -51,12 +110,46 @@ type goCollector struct { msMetrics memStatsMetrics } +const ( + // Those are not exposed due to need to move Go collector to another package in v2. + // See issue https://github.com/prometheus/client_golang/issues/1030. + goRuntimeMemStatsCollection uint32 = 1 << iota + goRuntimeMetricsCollection +) + +// GoCollectorOptions should not be used be directly by anything, except `collectors` package. +// Use it via collectors package instead. See issue +// https://github.com/prometheus/client_golang/issues/1030. +// +// Deprecated: Use collectors.WithGoCollections +type GoCollectorOptions struct { + // EnabledCollection sets what type of collections collector should expose on top of base collection. + // By default it's goMemStatsCollection | goRuntimeMetricsCollection. + EnabledCollections uint32 +} + +func (c GoCollectorOptions) isEnabled(flag uint32) bool { + return c.EnabledCollections&flag != 0 +} + +const defaultGoCollections = goRuntimeMemStatsCollection | goRuntimeMetricsCollection + // NewGoCollector is the obsolete version of collectors.NewGoCollector. // See there for documentation. // // Deprecated: Use collectors.NewGoCollector instead. -func NewGoCollector() Collector { - descriptions := metrics.All() +func NewGoCollector(opts ...func(o *GoCollectorOptions)) Collector { + opt := GoCollectorOptions{EnabledCollections: defaultGoCollections} + for _, o := range opts { + o(&opt) + } + + var descriptions []metrics.Description + if opt.isEnabled(goRuntimeMetricsCollection) { + descriptions = metrics.All() + } else if opt.isEnabled(goRuntimeMemStatsCollection) { + descriptions = bestEffortLookupRM(rmForMemStats) + } // Collect all histogram samples so that we can get their buckets. // The API guarantees that the buckets are always fixed for the lifetime @@ -67,7 +160,11 @@ func NewGoCollector() Collector { histograms = append(histograms, metrics.Sample{Name: d.Name}) } } - metrics.Read(histograms) + + if len(histograms) > 0 { + metrics.Read(histograms) + } + bucketsMap := make(map[string][]float64) for i := range histograms { bucketsMap[histograms[i].Name] = histograms[i].Value.Float64Histogram().Buckets @@ -83,7 +180,7 @@ func NewGoCollector() Collector { if !ok { // Just ignore this metric; we can't do anything with it here. // If a user decides to use the latest version of Go, we don't want - // to fail here. This condition is tested elsewhere. + // to fail here. This condition is tested in TestExpectedRuntimeMetrics. continue } @@ -123,12 +220,18 @@ func NewGoCollector() Collector { } metricSet = append(metricSet, m) } + + var msMetrics memStatsMetrics + if opt.isEnabled(goRuntimeMemStatsCollection) { + msMetrics = goRuntimeMemStats() + } return &goCollector{ + opt: opt, base: newBaseGoCollector(), rmSampleBuf: sampleBuf, rmSampleMap: sampleMap, rmMetrics: metricSet, - msMetrics: goRuntimeMemStats(), + msMetrics: msMetrics, } } @@ -163,40 +266,47 @@ func (c *goCollector) Collect(ch chan<- Metric) { c.mu.Lock() defer c.mu.Unlock() - // Populate runtime/metrics sample buffer. - metrics.Read(c.rmSampleBuf) + if len(c.rmSampleBuf) > 0 { + // Populate runtime/metrics sample buffer. + metrics.Read(c.rmSampleBuf) + } - // Update all our metrics from rmSampleBuf. - for i, sample := range c.rmSampleBuf { - // N.B. switch on concrete type because it's significantly more efficient - // than checking for the Counter and Gauge interface implementations. In - // this case, we control all the types here. - switch m := c.rmMetrics[i].(type) { - case *counter: - // Guard against decreases. This should never happen, but a failure - // to do so will result in a panic, which is a harsh consequence for - // a metrics collection bug. - v0, v1 := m.get(), unwrapScalarRMValue(sample.Value) - if v1 > v0 { - m.Add(unwrapScalarRMValue(sample.Value) - m.get()) + if c.opt.isEnabled(goRuntimeMetricsCollection) { + // Collect all our metrics from rmSampleBuf. + for i, sample := range c.rmSampleBuf { + // N.B. switch on concrete type because it's significantly more efficient + // than checking for the Counter and Gauge interface implementations. In + // this case, we control all the types here. + switch m := c.rmMetrics[i].(type) { + case *counter: + // Guard against decreases. This should never happen, but a failure + // to do so will result in a panic, which is a harsh consequence for + // a metrics collection bug. + v0, v1 := m.get(), unwrapScalarRMValue(sample.Value) + if v1 > v0 { + m.Add(unwrapScalarRMValue(sample.Value) - m.get()) + } + m.Collect(ch) + case *gauge: + m.Set(unwrapScalarRMValue(sample.Value)) + m.Collect(ch) + case *batchHistogram: + m.update(sample.Value.Float64Histogram(), c.exactSumFor(sample.Name)) + m.Collect(ch) + default: + panic("unexpected metric type") } - m.Collect(ch) - case *gauge: - m.Set(unwrapScalarRMValue(sample.Value)) - m.Collect(ch) - case *batchHistogram: - m.update(sample.Value.Float64Histogram(), c.exactSumFor(sample.Name)) - m.Collect(ch) - default: - panic("unexpected metric type") } } + // ms is a dummy MemStats that we populate ourselves so that we can - // populate the old metrics from it. - var ms runtime.MemStats - memStatsFromRM(&ms, c.rmSampleMap) - for _, i := range c.msMetrics { - ch <- MustNewConstMetric(i.desc, i.valType, i.eval(&ms)) + // populate the old metrics from it if goMemStatsCollection is enabled. + if c.opt.isEnabled(goRuntimeMemStatsCollection) { + var ms runtime.MemStats + memStatsFromRM(&ms, c.rmSampleMap) + for _, i := range c.msMetrics { + ch <- MustNewConstMetric(i.desc, i.valType, i.eval(&ms)) + } } } @@ -261,35 +371,30 @@ func memStatsFromRM(ms *runtime.MemStats, rm map[string]*metrics.Sample) { // while having Mallocs - Frees still represent a live object count. // Unfortunately, MemStats doesn't actually export a large allocation count, // so it's impossible to pull this number out directly. - tinyAllocs := lookupOrZero("/gc/heap/tiny/allocs:objects") - ms.Mallocs = lookupOrZero("/gc/heap/allocs:objects") + tinyAllocs - ms.Frees = lookupOrZero("/gc/heap/frees:objects") + tinyAllocs + tinyAllocs := lookupOrZero(goGCHeapTinyAllocsObjects) + ms.Mallocs = lookupOrZero(goGCHeapAllocsObjects) + tinyAllocs + ms.Frees = lookupOrZero(goGCHeapFreesObjects) + tinyAllocs - ms.TotalAlloc = lookupOrZero("/gc/heap/allocs:bytes") - ms.Sys = lookupOrZero("/memory/classes/total:bytes") + ms.TotalAlloc = lookupOrZero(goGCHeapAllocsBytes) + ms.Sys = lookupOrZero(goMemoryClassesTotalBytes) ms.Lookups = 0 // Already always zero. - ms.HeapAlloc = lookupOrZero("/memory/classes/heap/objects:bytes") + ms.HeapAlloc = lookupOrZero(goMemoryClassesHeapObjectsBytes) ms.Alloc = ms.HeapAlloc - ms.HeapInuse = ms.HeapAlloc + lookupOrZero("/memory/classes/heap/unused:bytes") - ms.HeapReleased = lookupOrZero("/memory/classes/heap/released:bytes") - ms.HeapIdle = ms.HeapReleased + lookupOrZero("/memory/classes/heap/free:bytes") + ms.HeapInuse = ms.HeapAlloc + lookupOrZero(goMemoryClassesHeapUnusedBytes) + ms.HeapReleased = lookupOrZero(goMemoryClassesHeapReleasedBytes) + ms.HeapIdle = ms.HeapReleased + lookupOrZero(goMemoryClassesHeapFreeBytes) ms.HeapSys = ms.HeapInuse + ms.HeapIdle - ms.HeapObjects = lookupOrZero("/gc/heap/objects:objects") - ms.StackInuse = lookupOrZero("/memory/classes/heap/stacks:bytes") - ms.StackSys = ms.StackInuse + lookupOrZero("/memory/classes/os-stacks:bytes") - ms.MSpanInuse = lookupOrZero("/memory/classes/metadata/mspan/inuse:bytes") - ms.MSpanSys = ms.MSpanInuse + lookupOrZero("/memory/classes/metadata/mspan/free:bytes") - ms.MCacheInuse = lookupOrZero("/memory/classes/metadata/mcache/inuse:bytes") - ms.MCacheSys = ms.MCacheInuse + lookupOrZero("/memory/classes/metadata/mcache/free:bytes") - ms.BuckHashSys = lookupOrZero("/memory/classes/profiling/buckets:bytes") - ms.GCSys = lookupOrZero("/memory/classes/metadata/other:bytes") - ms.OtherSys = lookupOrZero("/memory/classes/other:bytes") - ms.NextGC = lookupOrZero("/gc/heap/goal:bytes") - - // N.B. LastGC is omitted because runtime.GCStats already has this. - // See https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034 - // for more details. - ms.LastGC = 0 + ms.HeapObjects = lookupOrZero(goGCHeapObjects) + ms.StackInuse = lookupOrZero(goMemoryClassesHeapStacksBytes) + ms.StackSys = ms.StackInuse + lookupOrZero(goMemoryClassesOSStacksBytes) + ms.MSpanInuse = lookupOrZero(goMemoryClassesMetadataMSpanInuseBytes) + ms.MSpanSys = ms.MSpanInuse + lookupOrZero(goMemoryClassesMetadataMSPanFreeBytes) + ms.MCacheInuse = lookupOrZero(goMemoryClassesMetadataMCacheInuseBytes) + ms.MCacheSys = ms.MCacheInuse + lookupOrZero(goMemoryClassesMetadataMCacheFreeBytes) + ms.BuckHashSys = lookupOrZero(goMemoryClassesProfilingBucketsBytes) + ms.GCSys = lookupOrZero(goMemoryClassesMetadataOtherBytes) + ms.OtherSys = lookupOrZero(goMemoryClassesOtherBytes) + ms.NextGC = lookupOrZero(goGCHeapGoalBytes) // N.B. GCCPUFraction is intentionally omitted. This metric is not useful, // and often misleading due to the fact that it's an average over the lifetime diff --git a/prometheus/go_collector_go117_test.go b/prometheus/go_collector_latest_test.go similarity index 72% rename from prometheus/go_collector_go117_test.go rename to prometheus/go_collector_latest_test.go index 9c5218f..a7fcadd 100644 --- a/prometheus/go_collector_go117_test.go +++ b/prometheus/go_collector_latest_test.go @@ -28,78 +28,96 @@ import ( dto "github.com/prometheus/client_model/go" ) -func TestGoCollectorRuntimeMetrics(t *testing.T) { - metrics := collectGoMetrics(t) - - msChecklist := make(map[string]bool) - for _, m := range goRuntimeMemStats() { - msChecklist[m.desc.fqName] = false - } - - if len(metrics) == 0 { - t.Fatal("no metrics created by Collect") - } - - // Check a few specific metrics. - // - // Checking them all is somewhat pointless because the runtime/metrics - // metrics are going to shift underneath us. Also if we try to check - // against the runtime/metrics package in an automated fashion we're kind - // of missing the point, because we have to do all the same work the code - // has to do to perform the translation. Same for supporting old metric - // names (the best we can do here is make sure they're all accounted for). - var sysBytes, allocs float64 - for _, m := range metrics { - name := m.Desc().fqName - switch name { - case "go_memory_classes_total_bytes": - checkMemoryMetric(t, m, &sysBytes) - case "go_sys_bytes": - checkMemoryMetric(t, m, &sysBytes) - case "go_gc_heap_allocs_bytes_total": - checkMemoryMetric(t, m, &allocs) - case "go_alloc_bytes_total": - checkMemoryMetric(t, m, &allocs) - } - if present, ok := msChecklist[name]; ok { - if present { - t.Errorf("memstats metric %s found more than once", name) - } - msChecklist[name] = true - } - } - for name := range msChecklist { - if present := msChecklist[name]; !present { - t.Errorf("memstats metric %s not collected", name) - } +func TestRmForMemStats(t *testing.T) { + if got, want := len(bestEffortLookupRM(rmForMemStats)), len(rmForMemStats); got != want { + t.Errorf("got %d, want %d metrics", got, want) } } -func checkMemoryMetric(t *testing.T, m Metric, expValue *float64) { - t.Helper() +func expectedBaseMetrics() map[string]struct{} { + metrics := map[string]struct{}{} + b := newBaseGoCollector() + for _, m := range []string{ + b.gcDesc.fqName, + b.goInfoDesc.fqName, + b.goroutinesDesc.fqName, + b.gcLastTimeDesc.fqName, + b.threadsDesc.fqName, + } { + metrics[m] = struct{}{} + } + return metrics +} - pb := &dto.Metric{} - m.Write(pb) - var value float64 - if g := pb.GetGauge(); g != nil { - value = g.GetValue() - } else { - value = pb.GetCounter().GetValue() +func addExpectedRuntimeMemStats(metrics map[string]struct{}) map[string]struct{} { + for _, m := range goRuntimeMemStats() { + metrics[m.desc.fqName] = struct{}{} } - if value <= 0 { - t.Error("bad value for total memory") + return metrics +} + +func addExpectedRuntimeMetrics(metrics map[string]struct{}) map[string]struct{} { + for _, m := range expectedRuntimeMetrics { + metrics[m] = struct{}{} } - if *expValue == 0 { - *expValue = value - } else if value != *expValue { - t.Errorf("legacy metric and runtime/metrics metric do not match: want %d, got %d", int64(*expValue), int64(value)) + return metrics +} + +func TestGoCollector(t *testing.T) { + for _, tcase := range []struct { + collections uint32 + expectedFQNameSet map[string]struct{} + }{ + { + collections: 0, + expectedFQNameSet: expectedBaseMetrics(), + }, + { + collections: goRuntimeMemStatsCollection, + expectedFQNameSet: addExpectedRuntimeMemStats(expectedBaseMetrics()), + }, + { + collections: goRuntimeMetricsCollection, + expectedFQNameSet: addExpectedRuntimeMetrics(expectedBaseMetrics()), + }, + { + collections: goRuntimeMemStatsCollection | goRuntimeMetricsCollection, + expectedFQNameSet: addExpectedRuntimeMemStats(addExpectedRuntimeMetrics(expectedBaseMetrics())), + }, + } { + if ok := t.Run("", func(t *testing.T) { + goMetrics := collectGoMetrics(t, tcase.collections) + goMetricSet := make(map[string]Metric) + for _, m := range goMetrics { + goMetricSet[m.Desc().fqName] = m + } + + for i := range goMetrics { + name := goMetrics[i].Desc().fqName + + if _, ok := tcase.expectedFQNameSet[name]; !ok { + t.Errorf("found unpexpected metric %s", name) + continue + } + } + + // Now iterate over the expected metrics and look for removals. + for expectedName := range tcase.expectedFQNameSet { + if _, ok := goMetricSet[expectedName]; !ok { + t.Errorf("missing expected metric %s in collection", expectedName) + continue + } + } + }); !ok { + return + } } } var sink interface{} func TestBatchHistogram(t *testing.T) { - goMetrics := collectGoMetrics(t) + goMetrics := collectGoMetrics(t, defaultGoCollections) var mhist Metric for _, m := range goMetrics { @@ -126,7 +144,7 @@ func TestBatchHistogram(t *testing.T) { for i := 0; i < 100; i++ { sink = make([]byte, 128) } - collectGoMetrics(t) + collectGoMetrics(t, defaultGoCollections) for i, v := range hist.counts { if v != countsCopy[i] { t.Error("counts changed during new collection") @@ -175,10 +193,12 @@ func TestBatchHistogram(t *testing.T) { } } -func collectGoMetrics(t *testing.T) []Metric { +func collectGoMetrics(t *testing.T, enabledCollections uint32) []Metric { t.Helper() - c := NewGoCollector().(*goCollector) + c := NewGoCollector(func(o *GoCollectorOptions) { + o.EnabledCollections = enabledCollections + }).(*goCollector) // Collect all metrics. ch := make(chan Metric) @@ -201,7 +221,8 @@ func collectGoMetrics(t *testing.T) []Metric { func TestMemStatsEquivalence(t *testing.T) { var msReal, msFake runtime.MemStats - descs := metrics.All() + descs := bestEffortLookupRM(rmForMemStats) + samples := make([]metrics.Sample, len(descs)) samplesMap := make(map[string]*metrics.Sample) for i := range descs { @@ -214,9 +235,9 @@ func TestMemStatsEquivalence(t *testing.T) { // Populate msReal. runtime.ReadMemStats(&msReal) - - // Populate msFake. + // Populate msFake and hope that no GC happened in between (: metrics.Read(samples) + memStatsFromRM(&msFake, samplesMap) // Iterate over them and make sure they're somewhat close. @@ -227,9 +248,16 @@ func TestMemStatsEquivalence(t *testing.T) { for i := 0; i < msRealValue.NumField(); i++ { fr := msRealValue.Field(i) ff := msFakeValue.Field(i) - switch typ.Kind() { + + if typ.Field(i).Name == "PauseTotalNs" || typ.Field(i).Name == "LastGC" { + // We don't use those fields for metrics, + // thus we are not interested in having this filled. + continue + } + switch fr.Kind() { + // Fields which we are interested in are all uint64s. + // The only float64 field GCCPUFraction is by design omitted. case reflect.Uint64: - // N.B. Almost all fields of MemStats are uint64s. vr := fr.Interface().(uint64) vf := ff.Interface().(uint64) if float64(vr-vf)/float64(vf) > 0.05 { @@ -240,7 +268,7 @@ func TestMemStatsEquivalence(t *testing.T) { } func TestExpectedRuntimeMetrics(t *testing.T) { - goMetrics := collectGoMetrics(t) + goMetrics := collectGoMetrics(t, goRuntimeMetricsCollection) goMetricSet := make(map[string]Metric) for _, m := range goMetrics { goMetricSet[m.Desc().fqName] = m @@ -253,6 +281,7 @@ func TestExpectedRuntimeMetrics(t *testing.T) { rmName := descs[i].Name rmSet[rmName] = struct{}{} + // expectedRuntimeMetrics depends on Go version. expFQName, ok := expectedRuntimeMetrics[rmName] if !ok { t.Errorf("found new runtime/metrics metric %s", rmName) @@ -268,6 +297,7 @@ func TestExpectedRuntimeMetrics(t *testing.T) { continue } } + // Now iterate over the expected metrics and look for removals. cardinality := 0 for rmName, fqName := range expectedRuntimeMetrics { From 46d3dd4e6c6c89567048fc0e94a778802b0e09ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuel=20R=C3=BCger?= Date: Wed, 13 Apr 2022 17:21:42 +0200 Subject: [PATCH 22/37] Bump minimum required Go version to 1.16 (#1032) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Manuel Rüger --- .circleci/config.yml | 4 ---- CHANGELOG.md | 4 ++++ README.md | 2 +- go.mod | 2 +- prometheus/collectors/dbstats_collector_test.go | 5 +---- 5 files changed, 7 insertions(+), 10 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 1525318..d51e22d 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -46,10 +46,6 @@ workflows: client_golang: jobs: # Refer to README.md for the currently supported versions. - - test: - name: go-1-15 - go_version: "1.15" - run_lint: true - test: name: go-1-16 go_version: "1.16" diff --git a/CHANGELOG.md b/CHANGELOG.md index cf231ff..ac974ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,7 @@ +## Unreleased + +* [CHANGE] Minimum required Go version is now 1.16. + ## 1.12.1 / 2022-01-29 * [BUGFIX] Make the Go 1.17 collector concurrency-safe #969 diff --git a/README.md b/README.md index 0d09ba0..e401975 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ This is the [Go](http://golang.org) client library for instrumenting application code, and one for creating clients that talk to the Prometheus HTTP API. -__This library requires Go1.15 or later.__ +__This library requires Go1.16 or later.__ ## Important note about releases and stability diff --git a/go.mod b/go.mod index d0ace2d..f36810e 100644 --- a/go.mod +++ b/go.mod @@ -12,4 +12,4 @@ require ( google.golang.org/protobuf v1.28.0 ) -go 1.15 +go 1.16 diff --git a/prometheus/collectors/dbstats_collector_test.go b/prometheus/collectors/dbstats_collector_test.go index 4cfdda5..0698bb2 100644 --- a/prometheus/collectors/dbstats_collector_test.go +++ b/prometheus/collectors/dbstats_collector_test.go @@ -15,7 +15,6 @@ package collectors import ( "database/sql" - "runtime" "testing" "github.com/prometheus/client_golang/prometheus" @@ -50,9 +49,7 @@ func TestDBStatsCollector(t *testing.T) { "go_sql_wait_duration_seconds_total", "go_sql_max_idle_closed_total", "go_sql_max_lifetime_closed_total", - } - if runtime.Version() >= "go1.15" { - names = append(names, "go_sql_max_idle_time_closed_total") + "go_sql_max_idle_time_closed_total", } type result struct { found bool From 0c691ed35fb874d4b5996fb8390d05d7a204a6d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuel=20R=C3=BCger?= Date: Wed, 13 Apr 2022 19:35:14 +0200 Subject: [PATCH 23/37] go.mod: Exclude prometheus/client_golang v1.12.1 (#1027) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit See also https://github.com/prometheus/client_golang/issues/1012 As suggested in https://github.com/prometheus/client_golang/issues/1012#issuecomment-1090482644 Signed-off-by: Manuel Rüger --- go.mod | 3 +++ go.sum | 77 ++-------------------------------------------------------- 2 files changed, 5 insertions(+), 75 deletions(-) diff --git a/go.mod b/go.mod index f36810e..77f6115 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,7 @@ require ( github.com/beorn7/perks v1.0.1 github.com/cespare/xxhash/v2 v2.1.2 github.com/golang/protobuf v1.5.2 + github.com/jpillora/backoff v1.0.0 // indirect github.com/json-iterator/go v1.1.12 github.com/prometheus/client_model v0.2.0 github.com/prometheus/common v0.33.0 @@ -12,4 +13,6 @@ require ( google.golang.org/protobuf v1.28.0 ) +exclude github.com/prometheus/client_golang v1.12.1 + go 1.16 diff --git a/go.sum b/go.sum index df0996e..2f3a451 100644 --- a/go.sum +++ b/go.sum @@ -33,17 +33,9 @@ cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= -github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= -github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= -github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= -github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= -github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho= -github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= -github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= -github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE= github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= @@ -61,16 +53,8 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= -github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= -github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= -github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY= github.com/go-kit/log v0.2.0/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0= -github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= -github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= -github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= -github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= -github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= @@ -95,7 +79,6 @@ github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:W github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= -github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= @@ -129,19 +112,12 @@ github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/jpillora/backoff v1.0.0 h1:uvFg412JmmHBHw7iwprIxkPMI+sGQ4kzOWsMeHnm2EA= github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4= -github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= -github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= -github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= -github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= -github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= -github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= @@ -149,50 +125,23 @@ github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 h1:ZqeYNhU3OHLH3mGKHDcjJRFFRrJa6eAM5H+CtDdOsPc= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= -github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f h1:KUppIJq7/+SVif2QVs3tOP0zanoHgBEVAwHxUSIzRqU= github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= -github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= -github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= -github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= -github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= -github.com/prometheus/client_golang v1.12.1/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY= -github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= -github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M= github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= -github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= -github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc= -github.com/prometheus/common v0.32.1/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls= github.com/prometheus/common v0.33.0 h1:rHgav/0a6+uYgGdNt3jwz8FNSesO/Hsang3O0T9A5SE= github.com/prometheus/common v0.33.0/go.mod h1:gB3sOl7P0TvJabZpLY5uQMpUqRCPPCyRLCZYc7JZTNE= -github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= -github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= -github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= -github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA= github.com/prometheus/procfs v0.7.3 h1:4jVXhlkAyzOScmCkXBTOLRLTz8EeU+eyjrwB/EPq0VU= github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= -github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= -github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= -github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= @@ -204,7 +153,6 @@ go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= -golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -242,7 +190,6 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= @@ -250,7 +197,6 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= -golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= @@ -268,7 +214,6 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= -golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc= golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= @@ -277,7 +222,6 @@ golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4Iltr golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b h1:clP8eMhB30EHdc0bd2Twtq6kgU7yl5ub2cQLSdrv1Dg= golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -290,12 +234,9 @@ golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -304,7 +245,6 @@ golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -317,26 +257,18 @@ golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886 h1:eJv7u3ksNXoLbGSKuv2s/SIO4tJVxc/A+MTpzxDgz/Q= golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -467,15 +399,10 @@ google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscL google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= -gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= From cd90f33be86a5cc9a9fbe7d2a7e673a31845bf83 Mon Sep 17 00:00:00 2001 From: Sourik Ghosh <61813998+sourikghosh@users.noreply.github.com> Date: Wed, 13 Apr 2022 23:08:05 +0530 Subject: [PATCH 24/37] smart diff to testutil.GatherAndCompare (#998) * added smart diff to testutil.GatherAndCompare Signed-off-by: Sourik Ghosh * v2 testuitls added for better diff Signed-off-by: Sourik Ghosh * replaced CollectAndCompare to CollectAndCompareV2 in testutil_test Signed-off-by: Sourik Ghosh * renamed methods from v2 to withT Signed-off-by: Sourik Ghosh * replaced testify with custom diff func Signed-off-by: Sourik Ghosh * difflib GetUnifiedDiffString added with test Signed-off-by: Sourik Ghosh * license meta data added to file Signed-off-by: Sourik Ghosh * moved difflib to internal Signed-off-by: Sourik Ghosh --- go.mod | 1 + prometheus/internal/difflib.go | 649 +++++++++++++++++++++++++++ prometheus/internal/difflib_test.go | 266 +++++++++++ prometheus/testutil/testutil.go | 75 +++- prometheus/testutil/testutil_test.go | 21 +- 5 files changed, 992 insertions(+), 20 deletions(-) create mode 100644 prometheus/internal/difflib.go create mode 100644 prometheus/internal/difflib_test.go diff --git a/go.mod b/go.mod index 77f6115..5135421 100644 --- a/go.mod +++ b/go.mod @@ -3,6 +3,7 @@ module github.com/prometheus/client_golang require ( github.com/beorn7/perks v1.0.1 github.com/cespare/xxhash/v2 v2.1.2 + github.com/davecgh/go-spew v1.1.1 github.com/golang/protobuf v1.5.2 github.com/jpillora/backoff v1.0.0 // indirect github.com/json-iterator/go v1.1.12 diff --git a/prometheus/internal/difflib.go b/prometheus/internal/difflib.go new file mode 100644 index 0000000..1789006 --- /dev/null +++ b/prometheus/internal/difflib.go @@ -0,0 +1,649 @@ +// Copyright 2022 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// It provides tools to compare sequences of strings and generate textual diffs. +// +// Maintaining `GetUnifiedDiffString` here because original repository +// (https://github.com/pmezard/go-difflib) is no loger maintained. +package internal + +import ( + "bufio" + "bytes" + "fmt" + "io" + "strings" +) + +func min(a, b int) int { + if a < b { + return a + } + return b +} + +func max(a, b int) int { + if a > b { + return a + } + return b +} + +func calculateRatio(matches, length int) float64 { + if length > 0 { + return 2.0 * float64(matches) / float64(length) + } + return 1.0 +} + +type Match struct { + A int + B int + Size int +} + +type OpCode struct { + Tag byte + I1 int + I2 int + J1 int + J2 int +} + +// SequenceMatcher compares sequence of strings. The basic +// algorithm predates, and is a little fancier than, an algorithm +// published in the late 1980's by Ratcliff and Obershelp under the +// hyperbolic name "gestalt pattern matching". The basic idea is to find +// the longest contiguous matching subsequence that contains no "junk" +// elements (R-O doesn't address junk). The same idea is then applied +// recursively to the pieces of the sequences to the left and to the right +// of the matching subsequence. This does not yield minimal edit +// sequences, but does tend to yield matches that "look right" to people. +// +// SequenceMatcher tries to compute a "human-friendly diff" between two +// sequences. Unlike e.g. UNIX(tm) diff, the fundamental notion is the +// longest *contiguous* & junk-free matching subsequence. That's what +// catches peoples' eyes. The Windows(tm) windiff has another interesting +// notion, pairing up elements that appear uniquely in each sequence. +// That, and the method here, appear to yield more intuitive difference +// reports than does diff. This method appears to be the least vulnerable +// to synching up on blocks of "junk lines", though (like blank lines in +// ordinary text files, or maybe "

" lines in HTML files). That may be +// because this is the only method of the 3 that has a *concept* of +// "junk" . +// +// Timing: Basic R-O is cubic time worst case and quadratic time expected +// case. SequenceMatcher is quadratic time for the worst case and has +// expected-case behavior dependent in a complicated way on how many +// elements the sequences have in common; best case time is linear. +type SequenceMatcher struct { + a []string + b []string + b2j map[string][]int + IsJunk func(string) bool + autoJunk bool + bJunk map[string]struct{} + matchingBlocks []Match + fullBCount map[string]int + bPopular map[string]struct{} + opCodes []OpCode +} + +func NewMatcher(a, b []string) *SequenceMatcher { + m := SequenceMatcher{autoJunk: true} + m.SetSeqs(a, b) + return &m +} + +func NewMatcherWithJunk(a, b []string, autoJunk bool, + isJunk func(string) bool) *SequenceMatcher { + + m := SequenceMatcher{IsJunk: isJunk, autoJunk: autoJunk} + m.SetSeqs(a, b) + return &m +} + +// Set two sequences to be compared. +func (m *SequenceMatcher) SetSeqs(a, b []string) { + m.SetSeq1(a) + m.SetSeq2(b) +} + +// Set the first sequence to be compared. The second sequence to be compared is +// not changed. +// +// SequenceMatcher computes and caches detailed information about the second +// sequence, so if you want to compare one sequence S against many sequences, +// use .SetSeq2(s) once and call .SetSeq1(x) repeatedly for each of the other +// sequences. +// +// See also SetSeqs() and SetSeq2(). +func (m *SequenceMatcher) SetSeq1(a []string) { + if &a == &m.a { + return + } + m.a = a + m.matchingBlocks = nil + m.opCodes = nil +} + +// Set the second sequence to be compared. The first sequence to be compared is +// not changed. +func (m *SequenceMatcher) SetSeq2(b []string) { + if &b == &m.b { + return + } + m.b = b + m.matchingBlocks = nil + m.opCodes = nil + m.fullBCount = nil + m.chainB() +} + +func (m *SequenceMatcher) chainB() { + // Populate line -> index mapping + b2j := map[string][]int{} + for i, s := range m.b { + indices := b2j[s] + indices = append(indices, i) + b2j[s] = indices + } + + // Purge junk elements + m.bJunk = map[string]struct{}{} + if m.IsJunk != nil { + junk := m.bJunk + for s, _ := range b2j { + if m.IsJunk(s) { + junk[s] = struct{}{} + } + } + for s, _ := range junk { + delete(b2j, s) + } + } + + // Purge remaining popular elements + popular := map[string]struct{}{} + n := len(m.b) + if m.autoJunk && n >= 200 { + ntest := n/100 + 1 + for s, indices := range b2j { + if len(indices) > ntest { + popular[s] = struct{}{} + } + } + for s, _ := range popular { + delete(b2j, s) + } + } + m.bPopular = popular + m.b2j = b2j +} + +func (m *SequenceMatcher) isBJunk(s string) bool { + _, ok := m.bJunk[s] + return ok +} + +// Find longest matching block in a[alo:ahi] and b[blo:bhi]. +// +// If IsJunk is not defined: +// +// Return (i,j,k) such that a[i:i+k] is equal to b[j:j+k], where +// alo <= i <= i+k <= ahi +// blo <= j <= j+k <= bhi +// and for all (i',j',k') meeting those conditions, +// k >= k' +// i <= i' +// and if i == i', j <= j' +// +// In other words, of all maximal matching blocks, return one that +// starts earliest in a, and of all those maximal matching blocks that +// start earliest in a, return the one that starts earliest in b. +// +// If IsJunk is defined, first the longest matching block is +// determined as above, but with the additional restriction that no +// junk element appears in the block. Then that block is extended as +// far as possible by matching (only) junk elements on both sides. So +// the resulting block never matches on junk except as identical junk +// happens to be adjacent to an "interesting" match. +// +// If no blocks match, return (alo, blo, 0). +func (m *SequenceMatcher) findLongestMatch(alo, ahi, blo, bhi int) Match { + // CAUTION: stripping common prefix or suffix would be incorrect. + // E.g., + // ab + // acab + // Longest matching block is "ab", but if common prefix is + // stripped, it's "a" (tied with "b"). UNIX(tm) diff does so + // strip, so ends up claiming that ab is changed to acab by + // inserting "ca" in the middle. That's minimal but unintuitive: + // "it's obvious" that someone inserted "ac" at the front. + // Windiff ends up at the same place as diff, but by pairing up + // the unique 'b's and then matching the first two 'a's. + besti, bestj, bestsize := alo, blo, 0 + + // find longest junk-free match + // during an iteration of the loop, j2len[j] = length of longest + // junk-free match ending with a[i-1] and b[j] + j2len := map[int]int{} + for i := alo; i != ahi; i++ { + // look at all instances of a[i] in b; note that because + // b2j has no junk keys, the loop is skipped if a[i] is junk + newj2len := map[int]int{} + for _, j := range m.b2j[m.a[i]] { + // a[i] matches b[j] + if j < blo { + continue + } + if j >= bhi { + break + } + k := j2len[j-1] + 1 + newj2len[j] = k + if k > bestsize { + besti, bestj, bestsize = i-k+1, j-k+1, k + } + } + j2len = newj2len + } + + // Extend the best by non-junk elements on each end. In particular, + // "popular" non-junk elements aren't in b2j, which greatly speeds + // the inner loop above, but also means "the best" match so far + // doesn't contain any junk *or* popular non-junk elements. + for besti > alo && bestj > blo && !m.isBJunk(m.b[bestj-1]) && + m.a[besti-1] == m.b[bestj-1] { + besti, bestj, bestsize = besti-1, bestj-1, bestsize+1 + } + for besti+bestsize < ahi && bestj+bestsize < bhi && + !m.isBJunk(m.b[bestj+bestsize]) && + m.a[besti+bestsize] == m.b[bestj+bestsize] { + bestsize += 1 + } + + // Now that we have a wholly interesting match (albeit possibly + // empty!), we may as well suck up the matching junk on each + // side of it too. Can't think of a good reason not to, and it + // saves post-processing the (possibly considerable) expense of + // figuring out what to do with it. In the case of an empty + // interesting match, this is clearly the right thing to do, + // because no other kind of match is possible in the regions. + for besti > alo && bestj > blo && m.isBJunk(m.b[bestj-1]) && + m.a[besti-1] == m.b[bestj-1] { + besti, bestj, bestsize = besti-1, bestj-1, bestsize+1 + } + for besti+bestsize < ahi && bestj+bestsize < bhi && + m.isBJunk(m.b[bestj+bestsize]) && + m.a[besti+bestsize] == m.b[bestj+bestsize] { + bestsize += 1 + } + + return Match{A: besti, B: bestj, Size: bestsize} +} + +// Return list of triples describing matching subsequences. +// +// Each triple is of the form (i, j, n), and means that +// a[i:i+n] == b[j:j+n]. The triples are monotonically increasing in +// i and in j. It's also guaranteed that if (i, j, n) and (i', j', n') are +// adjacent triples in the list, and the second is not the last triple in the +// list, then i+n != i' or j+n != j'. IOW, adjacent triples never describe +// adjacent equal blocks. +// +// The last triple is a dummy, (len(a), len(b), 0), and is the only +// triple with n==0. +func (m *SequenceMatcher) GetMatchingBlocks() []Match { + if m.matchingBlocks != nil { + return m.matchingBlocks + } + + var matchBlocks func(alo, ahi, blo, bhi int, matched []Match) []Match + matchBlocks = func(alo, ahi, blo, bhi int, matched []Match) []Match { + match := m.findLongestMatch(alo, ahi, blo, bhi) + i, j, k := match.A, match.B, match.Size + if match.Size > 0 { + if alo < i && blo < j { + matched = matchBlocks(alo, i, blo, j, matched) + } + matched = append(matched, match) + if i+k < ahi && j+k < bhi { + matched = matchBlocks(i+k, ahi, j+k, bhi, matched) + } + } + return matched + } + matched := matchBlocks(0, len(m.a), 0, len(m.b), nil) + + // It's possible that we have adjacent equal blocks in the + // matching_blocks list now. + nonAdjacent := []Match{} + i1, j1, k1 := 0, 0, 0 + for _, b := range matched { + // Is this block adjacent to i1, j1, k1? + i2, j2, k2 := b.A, b.B, b.Size + if i1+k1 == i2 && j1+k1 == j2 { + // Yes, so collapse them -- this just increases the length of + // the first block by the length of the second, and the first + // block so lengthened remains the block to compare against. + k1 += k2 + } else { + // Not adjacent. Remember the first block (k1==0 means it's + // the dummy we started with), and make the second block the + // new block to compare against. + if k1 > 0 { + nonAdjacent = append(nonAdjacent, Match{i1, j1, k1}) + } + i1, j1, k1 = i2, j2, k2 + } + } + if k1 > 0 { + nonAdjacent = append(nonAdjacent, Match{i1, j1, k1}) + } + + nonAdjacent = append(nonAdjacent, Match{len(m.a), len(m.b), 0}) + m.matchingBlocks = nonAdjacent + return m.matchingBlocks +} + +// Return list of 5-tuples describing how to turn a into b. +// +// Each tuple is of the form (tag, i1, i2, j1, j2). The first tuple +// has i1 == j1 == 0, and remaining tuples have i1 == the i2 from the +// tuple preceding it, and likewise for j1 == the previous j2. +// +// The tags are characters, with these meanings: +// +// 'r' (replace): a[i1:i2] should be replaced by b[j1:j2] +// +// 'd' (delete): a[i1:i2] should be deleted, j1==j2 in this case. +// +// 'i' (insert): b[j1:j2] should be inserted at a[i1:i1], i1==i2 in this case. +// +// 'e' (equal): a[i1:i2] == b[j1:j2] +func (m *SequenceMatcher) GetOpCodes() []OpCode { + if m.opCodes != nil { + return m.opCodes + } + i, j := 0, 0 + matching := m.GetMatchingBlocks() + opCodes := make([]OpCode, 0, len(matching)) + for _, m := range matching { + // invariant: we've pumped out correct diffs to change + // a[:i] into b[:j], and the next matching block is + // a[ai:ai+size] == b[bj:bj+size]. So we need to pump + // out a diff to change a[i:ai] into b[j:bj], pump out + // the matching block, and move (i,j) beyond the match + ai, bj, size := m.A, m.B, m.Size + tag := byte(0) + if i < ai && j < bj { + tag = 'r' + } else if i < ai { + tag = 'd' + } else if j < bj { + tag = 'i' + } + if tag > 0 { + opCodes = append(opCodes, OpCode{tag, i, ai, j, bj}) + } + i, j = ai+size, bj+size + // the list of matching blocks is terminated by a + // sentinel with size 0 + if size > 0 { + opCodes = append(opCodes, OpCode{'e', ai, i, bj, j}) + } + } + m.opCodes = opCodes + return m.opCodes +} + +// Isolate change clusters by eliminating ranges with no changes. +// +// Return a generator of groups with up to n lines of context. +// Each group is in the same format as returned by GetOpCodes(). +func (m *SequenceMatcher) GetGroupedOpCodes(n int) [][]OpCode { + if n < 0 { + n = 3 + } + codes := m.GetOpCodes() + if len(codes) == 0 { + codes = []OpCode{{'e', 0, 1, 0, 1}} + } + // Fixup leading and trailing groups if they show no changes. + if codes[0].Tag == 'e' { + c := codes[0] + i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2 + codes[0] = OpCode{c.Tag, max(i1, i2-n), i2, max(j1, j2-n), j2} + } + if codes[len(codes)-1].Tag == 'e' { + c := codes[len(codes)-1] + i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2 + codes[len(codes)-1] = OpCode{c.Tag, i1, min(i2, i1+n), j1, min(j2, j1+n)} + } + nn := n + n + groups := [][]OpCode{} + group := []OpCode{} + for _, c := range codes { + i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2 + // End the current group and start a new one whenever + // there is a large range with no changes. + if c.Tag == 'e' && i2-i1 > nn { + group = append(group, OpCode{c.Tag, i1, min(i2, i1+n), + j1, min(j2, j1+n)}) + groups = append(groups, group) + group = []OpCode{} + i1, j1 = max(i1, i2-n), max(j1, j2-n) + } + group = append(group, OpCode{c.Tag, i1, i2, j1, j2}) + } + if len(group) > 0 && !(len(group) == 1 && group[0].Tag == 'e') { + groups = append(groups, group) + } + return groups +} + +// Return a measure of the sequences' similarity (float in [0,1]). +// +// Where T is the total number of elements in both sequences, and +// M is the number of matches, this is 2.0*M / T. +// Note that this is 1 if the sequences are identical, and 0 if +// they have nothing in common. +// +// .Ratio() is expensive to compute if you haven't already computed +// .GetMatchingBlocks() or .GetOpCodes(), in which case you may +// want to try .QuickRatio() or .RealQuickRation() first to get an +// upper bound. +func (m *SequenceMatcher) Ratio() float64 { + matches := 0 + for _, m := range m.GetMatchingBlocks() { + matches += m.Size + } + return calculateRatio(matches, len(m.a)+len(m.b)) +} + +// Return an upper bound on ratio() relatively quickly. +// +// This isn't defined beyond that it is an upper bound on .Ratio(), and +// is faster to compute. +func (m *SequenceMatcher) QuickRatio() float64 { + // viewing a and b as multisets, set matches to the cardinality + // of their intersection; this counts the number of matches + // without regard to order, so is clearly an upper bound + if m.fullBCount == nil { + m.fullBCount = map[string]int{} + for _, s := range m.b { + m.fullBCount[s] = m.fullBCount[s] + 1 + } + } + + // avail[x] is the number of times x appears in 'b' less the + // number of times we've seen it in 'a' so far ... kinda + avail := map[string]int{} + matches := 0 + for _, s := range m.a { + n, ok := avail[s] + if !ok { + n = m.fullBCount[s] + } + avail[s] = n - 1 + if n > 0 { + matches += 1 + } + } + return calculateRatio(matches, len(m.a)+len(m.b)) +} + +// Return an upper bound on ratio() very quickly. +// +// This isn't defined beyond that it is an upper bound on .Ratio(), and +// is faster to compute than either .Ratio() or .QuickRatio(). +func (m *SequenceMatcher) RealQuickRatio() float64 { + la, lb := len(m.a), len(m.b) + return calculateRatio(min(la, lb), la+lb) +} + +// Convert range to the "ed" format +func formatRangeUnified(start, stop int) string { + // Per the diff spec at http://www.unix.org/single_unix_specification/ + beginning := start + 1 // lines start numbering with one + length := stop - start + if length == 1 { + return fmt.Sprintf("%d", beginning) + } + if length == 0 { + beginning -= 1 // empty ranges begin at line just before the range + } + return fmt.Sprintf("%d,%d", beginning, length) +} + +// Unified diff parameters +type UnifiedDiff struct { + A []string // First sequence lines + FromFile string // First file name + FromDate string // First file time + B []string // Second sequence lines + ToFile string // Second file name + ToDate string // Second file time + Eol string // Headers end of line, defaults to LF + Context int // Number of context lines +} + +// Compare two sequences of lines; generate the delta as a unified diff. +// +// Unified diffs are a compact way of showing line changes and a few +// lines of context. The number of context lines is set by 'n' which +// defaults to three. +// +// By default, the diff control lines (those with ---, +++, or @@) are +// created with a trailing newline. This is helpful so that inputs +// created from file.readlines() result in diffs that are suitable for +// file.writelines() since both the inputs and outputs have trailing +// newlines. +// +// For inputs that do not have trailing newlines, set the lineterm +// argument to "" so that the output will be uniformly newline free. +// +// The unidiff format normally has a header for filenames and modification +// times. Any or all of these may be specified using strings for +// 'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'. +// The modification times are normally expressed in the ISO 8601 format. +func WriteUnifiedDiff(writer io.Writer, diff UnifiedDiff) error { + buf := bufio.NewWriter(writer) + defer buf.Flush() + wf := func(format string, args ...interface{}) error { + _, err := buf.WriteString(fmt.Sprintf(format, args...)) + return err + } + ws := func(s string) error { + _, err := buf.WriteString(s) + return err + } + + if len(diff.Eol) == 0 { + diff.Eol = "\n" + } + + started := false + m := NewMatcher(diff.A, diff.B) + for _, g := range m.GetGroupedOpCodes(diff.Context) { + if !started { + started = true + fromDate := "" + if len(diff.FromDate) > 0 { + fromDate = "\t" + diff.FromDate + } + toDate := "" + if len(diff.ToDate) > 0 { + toDate = "\t" + diff.ToDate + } + if diff.FromFile != "" || diff.ToFile != "" { + err := wf("--- %s%s%s", diff.FromFile, fromDate, diff.Eol) + if err != nil { + return err + } + err = wf("+++ %s%s%s", diff.ToFile, toDate, diff.Eol) + if err != nil { + return err + } + } + } + first, last := g[0], g[len(g)-1] + range1 := formatRangeUnified(first.I1, last.I2) + range2 := formatRangeUnified(first.J1, last.J2) + if err := wf("@@ -%s +%s @@%s", range1, range2, diff.Eol); err != nil { + return err + } + for _, c := range g { + i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2 + if c.Tag == 'e' { + for _, line := range diff.A[i1:i2] { + if err := ws(" " + line); err != nil { + return err + } + } + continue + } + if c.Tag == 'r' || c.Tag == 'd' { + for _, line := range diff.A[i1:i2] { + if err := ws("-" + line); err != nil { + return err + } + } + } + if c.Tag == 'r' || c.Tag == 'i' { + for _, line := range diff.B[j1:j2] { + if err := ws("+" + line); err != nil { + return err + } + } + } + } + } + return nil +} + +// Like WriteUnifiedDiff but returns the diff a string. +func GetUnifiedDiffString(diff UnifiedDiff) (string, error) { + w := &bytes.Buffer{} + err := WriteUnifiedDiff(w, diff) + return string(w.Bytes()), err +} + +// Split a string on "\n" while preserving them. The output can be used +// as input for UnifiedDiff and ContextDiff structures. +func SplitLines(s string) []string { + lines := strings.SplitAfter(s, "\n") + lines[len(lines)-1] += "\n" + return lines +} diff --git a/prometheus/internal/difflib_test.go b/prometheus/internal/difflib_test.go new file mode 100644 index 0000000..1a2bb34 --- /dev/null +++ b/prometheus/internal/difflib_test.go @@ -0,0 +1,266 @@ +// Copyright 2022 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +package internal + +import ( + "bytes" + "fmt" + "math" + "reflect" + "strings" + "testing" +) + +func assertAlmostEqual(t *testing.T, a, b float64, places int) { + if math.Abs(a-b) > math.Pow10(-places) { + t.Errorf("%.7f != %.7f", a, b) + } +} + +func assertEqual(t *testing.T, a, b interface{}) { + if !reflect.DeepEqual(a, b) { + t.Errorf("%v != %v", a, b) + } +} + +func splitChars(s string) []string { + chars := make([]string, 0, len(s)) + // Assume ASCII inputs + for i := 0; i != len(s); i++ { + chars = append(chars, string(s[i])) + } + return chars +} + +func TestSequenceMatcherRatio(t *testing.T) { + s := NewMatcher(splitChars("abcd"), splitChars("bcde")) + assertEqual(t, s.Ratio(), 0.75) + assertEqual(t, s.QuickRatio(), 0.75) + assertEqual(t, s.RealQuickRatio(), 1.0) +} + +func TestGetOptCodes(t *testing.T) { + a := "qabxcd" + b := "abycdf" + s := NewMatcher(splitChars(a), splitChars(b)) + w := &bytes.Buffer{} + for _, op := range s.GetOpCodes() { + fmt.Fprintf(w, "%s a[%d:%d], (%s) b[%d:%d] (%s)\n", string(op.Tag), + op.I1, op.I2, a[op.I1:op.I2], op.J1, op.J2, b[op.J1:op.J2]) + } + result := string(w.Bytes()) + expected := `d a[0:1], (q) b[0:0] () +e a[1:3], (ab) b[0:2] (ab) +r a[3:4], (x) b[2:3] (y) +e a[4:6], (cd) b[3:5] (cd) +i a[6:6], () b[5:6] (f) +` + if expected != result { + t.Errorf("unexpected op codes: \n%s", result) + } +} + +func TestGroupedOpCodes(t *testing.T) { + a := []string{} + for i := 0; i != 39; i++ { + a = append(a, fmt.Sprintf("%02d", i)) + } + b := []string{} + b = append(b, a[:8]...) + b = append(b, " i") + b = append(b, a[8:19]...) + b = append(b, " x") + b = append(b, a[20:22]...) + b = append(b, a[27:34]...) + b = append(b, " y") + b = append(b, a[35:]...) + s := NewMatcher(a, b) + w := &bytes.Buffer{} + for _, g := range s.GetGroupedOpCodes(-1) { + fmt.Fprintf(w, "group\n") + for _, op := range g { + fmt.Fprintf(w, " %s, %d, %d, %d, %d\n", string(op.Tag), + op.I1, op.I2, op.J1, op.J2) + } + } + result := string(w.Bytes()) + expected := `group + e, 5, 8, 5, 8 + i, 8, 8, 8, 9 + e, 8, 11, 9, 12 +group + e, 16, 19, 17, 20 + r, 19, 20, 20, 21 + e, 20, 22, 21, 23 + d, 22, 27, 23, 23 + e, 27, 30, 23, 26 +group + e, 31, 34, 27, 30 + r, 34, 35, 30, 31 + e, 35, 38, 31, 34 +` + if expected != result { + t.Errorf("unexpected op codes: \n%s", result) + } +} + +func ExampleGetUnifiedDiffCode() { + a := `one +two +three +four +fmt.Printf("%s,%T",a,b)` + b := `zero +one +three +four` + diff := UnifiedDiff{ + A: SplitLines(a), + B: SplitLines(b), + FromFile: "Original", + FromDate: "2005-01-26 23:30:50", + ToFile: "Current", + ToDate: "2010-04-02 10:20:52", + Context: 3, + } + result, _ := GetUnifiedDiffString(diff) + fmt.Println(strings.Replace(result, "\t", " ", -1)) + // Output: + // --- Original 2005-01-26 23:30:50 + // +++ Current 2010-04-02 10:20:52 + // @@ -1,5 +1,4 @@ + // +zero + // one + // -two + // three + // four + // -fmt.Printf("%s,%T",a,b) +} + +func rep(s string, count int) string { + return strings.Repeat(s, count) +} + +func TestWithAsciiOneInsert(t *testing.T) { + sm := NewMatcher(splitChars(rep("b", 100)), + splitChars("a"+rep("b", 100))) + assertAlmostEqual(t, sm.Ratio(), 0.995, 3) + assertEqual(t, sm.GetOpCodes(), + []OpCode{{'i', 0, 0, 0, 1}, {'e', 0, 100, 1, 101}}) + assertEqual(t, len(sm.bPopular), 0) + + sm = NewMatcher(splitChars(rep("b", 100)), + splitChars(rep("b", 50)+"a"+rep("b", 50))) + assertAlmostEqual(t, sm.Ratio(), 0.995, 3) + assertEqual(t, sm.GetOpCodes(), + []OpCode{{'e', 0, 50, 0, 50}, {'i', 50, 50, 50, 51}, {'e', 50, 100, 51, 101}}) + assertEqual(t, len(sm.bPopular), 0) +} + +func TestWithAsciiOnDelete(t *testing.T) { + sm := NewMatcher(splitChars(rep("a", 40)+"c"+rep("b", 40)), + splitChars(rep("a", 40)+rep("b", 40))) + assertAlmostEqual(t, sm.Ratio(), 0.994, 3) + assertEqual(t, sm.GetOpCodes(), + []OpCode{{'e', 0, 40, 0, 40}, {'d', 40, 41, 40, 40}, {'e', 41, 81, 40, 80}}) +} + +func TestWithAsciiBJunk(t *testing.T) { + isJunk := func(s string) bool { + return s == " " + } + sm := NewMatcherWithJunk(splitChars(rep("a", 40)+rep("b", 40)), + splitChars(rep("a", 44)+rep("b", 40)), true, isJunk) + assertEqual(t, sm.bJunk, map[string]struct{}{}) + + sm = NewMatcherWithJunk(splitChars(rep("a", 40)+rep("b", 40)), + splitChars(rep("a", 44)+rep("b", 40)+rep(" ", 20)), false, isJunk) + assertEqual(t, sm.bJunk, map[string]struct{}{" ": struct{}{}}) + + isJunk = func(s string) bool { + return s == " " || s == "b" + } + sm = NewMatcherWithJunk(splitChars(rep("a", 40)+rep("b", 40)), + splitChars(rep("a", 44)+rep("b", 40)+rep(" ", 20)), false, isJunk) + assertEqual(t, sm.bJunk, map[string]struct{}{" ": struct{}{}, "b": struct{}{}}) +} + +func TestSFBugsRatioForNullSeqn(t *testing.T) { + sm := NewMatcher(nil, nil) + assertEqual(t, sm.Ratio(), 1.0) + assertEqual(t, sm.QuickRatio(), 1.0) + assertEqual(t, sm.RealQuickRatio(), 1.0) +} + +func TestSFBugsComparingEmptyLists(t *testing.T) { + groups := NewMatcher(nil, nil).GetGroupedOpCodes(-1) + assertEqual(t, len(groups), 0) + diff := UnifiedDiff{ + FromFile: "Original", + ToFile: "Current", + Context: 3, + } + result, err := GetUnifiedDiffString(diff) + assertEqual(t, err, nil) + assertEqual(t, result, "") +} + +func TestOutputFormatRangeFormatUnified(t *testing.T) { + // Per the diff spec at http://www.unix.org/single_unix_specification/ + // + // Each field shall be of the form: + // %1d", if the range contains exactly one line, + // and: + // "%1d,%1d", , otherwise. + // If a range is empty, its beginning line number shall be the number of + // the line just before the range, or 0 if the empty range starts the file. + fm := formatRangeUnified + assertEqual(t, fm(3, 3), "3,0") + assertEqual(t, fm(3, 4), "4") + assertEqual(t, fm(3, 5), "4,2") + assertEqual(t, fm(3, 6), "4,3") + assertEqual(t, fm(0, 0), "0,0") +} + +func TestSplitLines(t *testing.T) { + allTests := []struct { + input string + want []string + }{ + {"foo", []string{"foo\n"}}, + {"foo\nbar", []string{"foo\n", "bar\n"}}, + {"foo\nbar\n", []string{"foo\n", "bar\n", "\n"}}, + } + for _, test := range allTests { + assertEqual(t, SplitLines(test.input), test.want) + } +} + +func benchmarkSplitLines(b *testing.B, count int) { + str := strings.Repeat("foo\n", count) + + b.ResetTimer() + + n := 0 + for i := 0; i < b.N; i++ { + n += len(SplitLines(str)) + } +} + +func BenchmarkSplitLines100(b *testing.B) { + benchmarkSplitLines(b, 100) +} + +func BenchmarkSplitLines10000(b *testing.B) { + benchmarkSplitLines(b, 10000) +} diff --git a/prometheus/testutil/testutil.go b/prometheus/testutil/testutil.go index bf95bea..115979d 100644 --- a/prometheus/testutil/testutil.go +++ b/prometheus/testutil/testutil.go @@ -41,7 +41,9 @@ import ( "bytes" "fmt" "io" + "reflect" + "github.com/davecgh/go-spew/spew" "github.com/prometheus/common/expfmt" dto "github.com/prometheus/client_model/go" @@ -211,20 +213,73 @@ func compare(got, want []*dto.MetricFamily) error { return fmt.Errorf("encoding expected metrics failed: %s", err) } } - - if wantBuf.String() != gotBuf.String() { - return fmt.Errorf(` -metric output does not match expectation; want: - -%s -got: - -%s`, wantBuf.String(), gotBuf.String()) - + if diffErr := diff(wantBuf, gotBuf); diffErr != "" { + return fmt.Errorf(diffErr) } return nil } +// diff returns a diff of both values as long as both are of the same type and +// are a struct, map, slice, array or string. Otherwise it returns an empty string. +func diff(expected interface{}, actual interface{}) string { + if expected == nil || actual == nil { + return "" + } + + et, ek := typeAndKind(expected) + at, _ := typeAndKind(actual) + if et != at { + return "" + } + + if ek != reflect.Struct && ek != reflect.Map && ek != reflect.Slice && ek != reflect.Array && ek != reflect.String { + return "" + } + + var e, a string + c := spew.ConfigState{ + Indent: " ", + DisablePointerAddresses: true, + DisableCapacities: true, + SortKeys: true, + } + if et != reflect.TypeOf("") { + e = c.Sdump(expected) + a = c.Sdump(actual) + } else { + e = reflect.ValueOf(expected).String() + a = reflect.ValueOf(actual).String() + } + + diff, _ := internal.GetUnifiedDiffString(internal.UnifiedDiff{ + A: internal.SplitLines(e), + B: internal.SplitLines(a), + FromFile: "metric output does not match expectation; want", + FromDate: "", + ToFile: "got:", + ToDate: "", + Context: 1, + }) + + if diff == "" { + return "" + } + + return "\n\nDiff:\n" + diff +} + +// typeAndKind returns the type and kind of the given interface{} +func typeAndKind(v interface{}) (reflect.Type, reflect.Kind) { + t := reflect.TypeOf(v) + k := t.Kind() + + if k == reflect.Ptr { + t = t.Elem() + k = t.Kind() + } + return t, k +} + func filterMetrics(metrics []*dto.MetricFamily, names []string) []*dto.MetricFamily { var filtered []*dto.MetricFamily for _, m := range metrics { diff --git a/prometheus/testutil/testutil_test.go b/prometheus/testutil/testutil_test.go index 56d9933..96a6f38 100644 --- a/prometheus/testutil/testutil_test.go +++ b/prometheus/testutil/testutil_test.go @@ -284,17 +284,18 @@ func TestMetricNotFound(t *testing.T) { ` expectedError := ` -metric output does not match expectation; want: -# HELP some_other_metric A value that represents a counter. -# TYPE some_other_metric counter -some_other_metric{label1="value1"} 1 - -got: - -# HELP some_total A value that represents a counter. -# TYPE some_total counter -some_total{label1="value1"} 1 +Diff: +--- metric output does not match expectation; want ++++ got: +@@ -1,4 +1,4 @@ +-(bytes.Buffer) # HELP some_other_metric A value that represents a counter. +-# TYPE some_other_metric counter +-some_other_metric{label1="value1"} 1 ++(bytes.Buffer) # HELP some_total A value that represents a counter. ++# TYPE some_total counter ++some_total{label1="value1"} 1 + ` err := CollectAndCompare(c, strings.NewReader(metadata+expected)) From 11ee9add275a3290b24d7b55fe3f22c237d85a75 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 13 Apr 2022 20:43:29 +0200 Subject: [PATCH 25/37] gocollector: Reverted client_golang v1.12 addition of runtime/metrics metrics by default. (#1033) Fixes https://github.com/prometheus/client_golang/issues/967 Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 2 ++ prometheus/collectors/go_collector_latest.go | 4 ++-- prometheus/go_collector_latest.go | 2 +- prometheus/go_collector_latest_test.go | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac974ce..d515e69 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,8 @@ ## Unreleased * [CHANGE] Minimum required Go version is now 1.16. +* [CHANGE] Added `collectors.WithGoCollections` that allows to choose what collection of Go runtime metrics user wants: Equivalent of [`MemStats` structure](https://pkg.go.dev/runtime#MemStats) configured using `GoRuntimeMemStatsCollection`, new based on dedicated [runtime/metrics](https://pkg.go.dev/runtime/metrics) metrics represented by `GoRuntimeMetricsCollection` option, or both by specifying `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` flag. +* [CHANGE] :warning: Change in `collectors.NewGoCollector` metrics: Reverting addition of new ~80 runtime metrics by default. You can enable this back with `GoRuntimeMetricsCollection` option or `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` for smooth transition. ## 1.12.1 / 2022-01-29 diff --git a/prometheus/collectors/go_collector_latest.go b/prometheus/collectors/go_collector_latest.go index a4657a4..58b0a5b 100644 --- a/prometheus/collectors/go_collector_latest.go +++ b/prometheus/collectors/go_collector_latest.go @@ -70,8 +70,8 @@ const ( // WithGoCollections(GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection) means both GoRuntimeMemStatsCollection // metrics and GoRuntimeMetricsCollection will be exposed. // -// Use WithGoCollections(GoRuntimeMemStatsCollection) to have Go collector working in -// the compatibility mode with client_golang pre v1.12 (move to runtime/metrics). +// The current default is GoRuntimeMemStatsCollection, so the compatibility mode with +// client_golang pre v1.12 (move to runtime/metrics). func WithGoCollections(flags uint32) goOption { return func(o *goOptions) { o.EnabledCollections = flags diff --git a/prometheus/go_collector_latest.go b/prometheus/go_collector_latest.go index 944794f..8528ea7 100644 --- a/prometheus/go_collector_latest.go +++ b/prometheus/go_collector_latest.go @@ -132,7 +132,7 @@ func (c GoCollectorOptions) isEnabled(flag uint32) bool { return c.EnabledCollections&flag != 0 } -const defaultGoCollections = goRuntimeMemStatsCollection | goRuntimeMetricsCollection +const defaultGoCollections = goRuntimeMemStatsCollection // NewGoCollector is the obsolete version of collectors.NewGoCollector. // See there for documentation. diff --git a/prometheus/go_collector_latest_test.go b/prometheus/go_collector_latest_test.go index a7fcadd..88158df 100644 --- a/prometheus/go_collector_latest_test.go +++ b/prometheus/go_collector_latest_test.go @@ -117,7 +117,7 @@ func TestGoCollector(t *testing.T) { var sink interface{} func TestBatchHistogram(t *testing.T) { - goMetrics := collectGoMetrics(t, defaultGoCollections) + goMetrics := collectGoMetrics(t, goRuntimeMetricsCollection) var mhist Metric for _, m := range goMetrics { From 48a686a6033f5f500259d8acf1618bf76b11ddfe Mon Sep 17 00:00:00 2001 From: Joseph Woodward Date: Thu, 21 Apr 2022 06:23:16 +0100 Subject: [PATCH 26/37] Update query API to support timeouts (#1014) * Add timeout parameter for queries Signed-off-by: Joseph Woodward * Update api/prometheus/v1/api.go Co-authored-by: Kemal Akkoyun Signed-off-by: Joseph Woodward * Update api/prometheus/v1/api.go Co-authored-by: Kemal Akkoyun Signed-off-by: Joseph Woodward * Pass timeout as stringified time.Duration instead of millisecond value Signed-off-by: Joseph Woodward * Update QueryRange API to support timeouts Signed-off-by: Joseph Woodward * Add timeout to test request params Signed-off-by: Joseph Woodward Co-authored-by: Kemal Akkoyun --- api/prometheus/v1/api.go | 41 ++++++++++++++++++++++++++++--- api/prometheus/v1/api_test.go | 26 +++++++++++--------- api/prometheus/v1/example_test.go | 4 +-- 3 files changed, 53 insertions(+), 18 deletions(-) diff --git a/api/prometheus/v1/api.go b/api/prometheus/v1/api.go index 5ed0912..83b0efe 100644 --- a/api/prometheus/v1/api.go +++ b/api/prometheus/v1/api.go @@ -238,9 +238,9 @@ type API interface { // LabelValues performs a query for the values of the given label, time range and matchers. LabelValues(ctx context.Context, label string, matches []string, startTime time.Time, endTime time.Time) (model.LabelValues, Warnings, error) // Query performs a query for the given time. - Query(ctx context.Context, query string, ts time.Time) (model.Value, Warnings, error) + Query(ctx context.Context, query string, ts time.Time, opts ...Option) (model.Value, Warnings, error) // QueryRange performs a query for the given range. - QueryRange(ctx context.Context, query string, r Range) (model.Value, Warnings, error) + QueryRange(ctx context.Context, query string, r Range, opts ...Option) (model.Value, Warnings, error) // QueryExemplars performs a query for exemplars by the given query and time range. QueryExemplars(ctx context.Context, query string, startTime time.Time, endTime time.Time) ([]ExemplarQueryResult, error) // Buildinfo returns various build information properties about the Prometheus server @@ -818,10 +818,33 @@ func (h *httpAPI) LabelValues(ctx context.Context, label string, matches []strin return labelValues, w, json.Unmarshal(body, &labelValues) } -func (h *httpAPI) Query(ctx context.Context, query string, ts time.Time) (model.Value, Warnings, error) { +type apiOptions struct { + timeout time.Duration +} + +type Option func(c *apiOptions) + +func WithTimeout(timeout time.Duration) Option { + return func(o *apiOptions) { + o.timeout = timeout + } +} + +func (h *httpAPI) Query(ctx context.Context, query string, ts time.Time, opts ...Option) (model.Value, Warnings, error) { + u := h.client.URL(epQuery, nil) q := u.Query() + opt := &apiOptions{} + for _, o := range opts { + o(opt) + } + + d := opt.timeout + if d > 0 { + q.Set("timeout", d.String()) + } + q.Set("query", query) if !ts.IsZero() { q.Set("time", formatTime(ts)) @@ -836,7 +859,7 @@ func (h *httpAPI) Query(ctx context.Context, query string, ts time.Time) (model. return model.Value(qres.v), warnings, json.Unmarshal(body, &qres) } -func (h *httpAPI) QueryRange(ctx context.Context, query string, r Range) (model.Value, Warnings, error) { +func (h *httpAPI) QueryRange(ctx context.Context, query string, r Range, opts ...Option) (model.Value, Warnings, error) { u := h.client.URL(epQueryRange, nil) q := u.Query() @@ -845,6 +868,16 @@ func (h *httpAPI) QueryRange(ctx context.Context, query string, r Range) (model. q.Set("end", formatTime(r.End)) q.Set("step", strconv.FormatFloat(r.Step.Seconds(), 'f', -1, 64)) + opt := &apiOptions{} + for _, o := range opts { + o(opt) + } + + d := opt.timeout + if d > 0 { + q.Set("timeout", d.String()) + } + _, body, warnings, err := h.client.DoGetFallback(ctx, u, q) if err != nil { return nil, warnings, err diff --git a/api/prometheus/v1/api_test.go b/api/prometheus/v1/api_test.go index e7d0b4c..82c234b 100644 --- a/api/prometheus/v1/api_test.go +++ b/api/prometheus/v1/api_test.go @@ -170,15 +170,15 @@ func TestAPIs(t *testing.T) { } } - doQuery := func(q string, ts time.Time) func() (interface{}, Warnings, error) { + doQuery := func(q string, ts time.Time, opts ...Option) func() (interface{}, Warnings, error) { return func() (interface{}, Warnings, error) { - return promAPI.Query(context.Background(), q, ts) + return promAPI.Query(context.Background(), q, ts, opts...) } } - doQueryRange := func(q string, rng Range) func() (interface{}, Warnings, error) { + doQueryRange := func(q string, rng Range, opts ...Option) func() (interface{}, Warnings, error) { return func() (interface{}, Warnings, error) { - return promAPI.QueryRange(context.Background(), q, rng) + return promAPI.QueryRange(context.Background(), q, rng, opts...) } } @@ -246,7 +246,7 @@ func TestAPIs(t *testing.T) { queryTests := []apiTest{ { - do: doQuery("2", testTime), + do: doQuery("2", testTime, WithTimeout(5*time.Second)), inRes: &queryResult{ Type: model.ValScalar, Result: &model.Scalar{ @@ -258,8 +258,9 @@ func TestAPIs(t *testing.T) { reqMethod: "POST", reqPath: "/api/v1/query", reqParam: url.Values{ - "query": []string{"2"}, - "time": []string{testTime.Format(time.RFC3339Nano)}, + "query": []string{"2"}, + "time": []string{testTime.Format(time.RFC3339Nano)}, + "timeout": []string{(5 * time.Second).String()}, }, res: &model.Scalar{ Value: 2, @@ -365,16 +366,17 @@ func TestAPIs(t *testing.T) { Start: testTime.Add(-time.Minute), End: testTime, Step: time.Minute, - }), + }, WithTimeout(5*time.Second)), inErr: fmt.Errorf("some error"), reqMethod: "POST", reqPath: "/api/v1/query_range", reqParam: url.Values{ - "query": []string{"2"}, - "start": []string{testTime.Add(-time.Minute).Format(time.RFC3339Nano)}, - "end": []string{testTime.Format(time.RFC3339Nano)}, - "step": []string{time.Minute.String()}, + "query": []string{"2"}, + "start": []string{testTime.Add(-time.Minute).Format(time.RFC3339Nano)}, + "end": []string{testTime.Format(time.RFC3339Nano)}, + "step": []string{time.Minute.String()}, + "timeout": []string{(5 * time.Second).String()}, }, err: fmt.Errorf("some error"), }, diff --git a/api/prometheus/v1/example_test.go b/api/prometheus/v1/example_test.go index 8182902..f0ee3b6 100644 --- a/api/prometheus/v1/example_test.go +++ b/api/prometheus/v1/example_test.go @@ -39,7 +39,7 @@ func ExampleAPI_query() { v1api := v1.NewAPI(client) ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() - result, warnings, err := v1api.Query(ctx, "up", time.Now()) + result, warnings, err := v1api.Query(ctx, "up", time.Now(), v1.WithTimeout(5*time.Second)) if err != nil { fmt.Printf("Error querying Prometheus: %v\n", err) os.Exit(1) @@ -67,7 +67,7 @@ func ExampleAPI_queryRange() { End: time.Now(), Step: time.Minute, } - result, warnings, err := v1api.QueryRange(ctx, "rate(prometheus_tsdb_head_samples_appended_total[5m])", r) + result, warnings, err := v1api.QueryRange(ctx, "rate(prometheus_tsdb_head_samples_appended_total[5m])", r, v1.WithTimeout(5*time.Second)) if err != nil { fmt.Printf("Error querying Prometheus: %v\n", err) os.Exit(1) From 4dcf02ec7b3ccf640085e1013e1a95ca6ae221ea Mon Sep 17 00:00:00 2001 From: Zach Stone Date: Thu, 21 Apr 2022 08:29:05 +0200 Subject: [PATCH 27/37] Implement deletion based on partially matching labels (#1013) * WIP partial match Signed-off-by: Zach Stone * Cleanup Signed-off-by: Zach Stone * Comments Signed-off-by: Zach Stone * Tests and comments Signed-off-by: Zach Stone * Handle properly deleting multiple metrics, update tests Signed-off-by: Zach Stone * Comments Signed-off-by: Zach Stone * Try using curry values Signed-off-by: Zach Stone * Skip curry value to demo Signed-off-by: Zach Stone * Fix curry deletion, remove outdated comment. Signed-off-by: Zach Stone * Fix logic for deletion of metrics from prior to currying Signed-off-by: Zach Stone * Don't match curried values. Update tests. Signed-off-by: Zach Stone * Remove unneccesasry helper and todo comments Signed-off-by: Zach Stone * Comment about partial matching curried labels Signed-off-by: Zach Stone * Simplify curried value check Signed-off-by: Zach Stone --- prometheus/vec.go | 86 +++++++++++++++++++++++++++++++++++++++++ prometheus/vec_test.go | 87 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+) diff --git a/prometheus/vec.go b/prometheus/vec.go index 4ababe6..d0b5f0b 100644 --- a/prometheus/vec.go +++ b/prometheus/vec.go @@ -99,6 +99,16 @@ func (m *MetricVec) Delete(labels Labels) bool { return m.metricMap.deleteByHashWithLabels(h, labels, m.curry) } +// DeletePartialMatch deletes all metrics where the variable labels contain all of those +// passed in as labels. The order of the labels does not matter. +// It returns the number of metrics deleted. +// +// Note that curried labels will never be matched if deleting from the curried vector. +// To match curried labels with DeletePartialMatch, it must be called on the base vector. +func (m *MetricVec) DeletePartialMatch(labels Labels) int { + return m.metricMap.deleteByLabels(labels, m.curry) +} + // Without explicit forwarding of Describe, Collect, Reset, those methods won't // show up in GoDoc. @@ -381,6 +391,82 @@ func (m *metricMap) deleteByHashWithLabels( return true } +// deleteByLabels deletes a metric if the given labels are present in the metric. +func (m *metricMap) deleteByLabels(labels Labels, curry []curriedLabelValue) int { + m.mtx.Lock() + defer m.mtx.Unlock() + + var numDeleted int + + for h, metrics := range m.metrics { + i := findMetricWithPartialLabels(m.desc, metrics, labels, curry) + if i >= len(metrics) { + // Didn't find matching labels in this metric slice. + continue + } + delete(m.metrics, h) + numDeleted++ + } + + return numDeleted +} + +// findMetricWithPartialLabel returns the index of the matching metric or +// len(metrics) if not found. +func findMetricWithPartialLabels( + desc *Desc, metrics []metricWithLabelValues, labels Labels, curry []curriedLabelValue, +) int { + for i, metric := range metrics { + if matchPartialLabels(desc, metric.values, labels, curry) { + return i + } + } + return len(metrics) +} + +// indexOf searches the given slice of strings for the target string and returns +// the index or len(items) as well as a boolean whether the search succeeded. +func indexOf(target string, items []string) (int, bool) { + for i, l := range items { + if l == target { + return i, true + } + } + return len(items), false +} + +// valueMatchesVariableOrCurriedValue determines if a value was previously curried, +// and returns whether it matches either the "base" value or the curried value accordingly. +// It also indicates whether the match is against a curried or uncurried value. +func valueMatchesVariableOrCurriedValue(targetValue string, index int, values []string, curry []curriedLabelValue) (bool, bool) { + for _, curriedValue := range curry { + if curriedValue.index == index { + // This label was curried. See if the curried value matches our target. + return curriedValue.value == targetValue, true + } + } + // This label was not curried. See if the current value matches our target label. + return values[index] == targetValue, false +} + +// matchPartialLabels searches the current metric and returns whether all of the target label:value pairs are present. +func matchPartialLabels(desc *Desc, values []string, labels Labels, curry []curriedLabelValue) bool { + for l, v := range labels { + // Check if the target label exists in our metrics and get the index. + varLabelIndex, validLabel := indexOf(l, desc.variableLabels) + if validLabel { + // Check the value of that label against the target value. + // We don't consider curried values in partial matches. + matches, curried := valueMatchesVariableOrCurriedValue(v, varLabelIndex, values, curry) + if matches && !curried { + continue + } + } + return false + } + return true +} + // getOrCreateMetricWithLabelValues retrieves the metric by hash and label value // or creates it and returns the new one. // diff --git a/prometheus/vec_test.go b/prometheus/vec_test.go index bd18a9f..fdb050f 100644 --- a/prometheus/vec_test.go +++ b/prometheus/vec_test.go @@ -125,6 +125,93 @@ func testDeleteLabelValues(t *testing.T, vec *GaugeVec) { } } +func TestDeletePartialMatch(t *testing.T) { + baseVec := NewGaugeVec( + GaugeOpts{ + Name: "test", + Help: "helpless", + }, + []string{"l1", "l2", "l3"}, + ) + + assertNoMetric := func(t *testing.T) { + if n := len(baseVec.metricMap.metrics); n != 0 { + t.Error("expected no metrics, got", n) + } + } + + // No metric value is set. + if got, want := baseVec.DeletePartialMatch(Labels{"l1": "v1", "l2": "v2"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + baseVec.With(Labels{"l1": "baseValue1", "l2": "baseValue2", "l3": "baseValue3"}).Inc() + baseVec.With(Labels{"l1": "multiDeleteV1", "l2": "diff1BaseValue2", "l3": "v3"}).(Gauge).Set(42) + baseVec.With(Labels{"l1": "multiDeleteV1", "l2": "diff2BaseValue2", "l3": "v3"}).(Gauge).Set(84) + baseVec.With(Labels{"l1": "multiDeleteV1", "l2": "diff3BaseValue2", "l3": "v3"}).(Gauge).Set(168) + + curriedVec := baseVec.MustCurryWith(Labels{"l2": "curriedValue2"}) + curriedVec.WithLabelValues("curriedValue1", "curriedValue3").Inc() + curriedVec.WithLabelValues("curriedValue1", "differentCurriedValue3").Inc() + curriedVec.WithLabelValues("differentCurriedValue1", "differentCurriedValue3").Inc() + + // Try to delete nonexistent label with existent value from curried vector. + if got, want := curriedVec.DeletePartialMatch(Labels{"lx": "curriedValue1"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Try to delete valid label with nonexistent value from curried vector. + if got, want := curriedVec.DeletePartialMatch(Labels{"l1": "badValue1"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Try to delete from a curried vector based on labels which were curried. + // This operation succeeds when run against the base vector below. + if got, want := curriedVec.DeletePartialMatch(Labels{"l2": "curriedValue2"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Try to delete from a curried vector based on labels which were curried, + // but the value actually exists in the base vector. + if got, want := curriedVec.DeletePartialMatch(Labels{"l2": "baseValue2"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Delete multiple matching metrics from a curried vector based on partial values. + if got, want := curriedVec.DeletePartialMatch(Labels{"l1": "curriedValue1"}), 2; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Try to delete nonexistent label with existent value from base vector. + if got, want := baseVec.DeletePartialMatch(Labels{"lx": "curriedValue1"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Try to delete partially invalid labels from base vector. + if got, want := baseVec.DeletePartialMatch(Labels{"l1": "baseValue1", "l2": "badValue2"}), 0; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Delete from the base vector based on values which were curried. + // This operation fails when run against a curried vector above. + if got, want := baseVec.DeletePartialMatch(Labels{"l2": "curriedValue2"}), 1; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Delete multiple metrics from the base vector based on a single valid label. + if got, want := baseVec.DeletePartialMatch(Labels{"l1": "multiDeleteV1"}), 3; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // Delete from the base vector based on values which were not curried. + if got, want := baseVec.DeletePartialMatch(Labels{"l3": "baseValue3"}), 1; got != want { + t.Errorf("got %v, want %v", got, want) + } + + // All metrics should have been deleted now. + assertNoMetric(t) +} + func TestMetricVec(t *testing.T) { vec := NewGaugeVec( GaugeOpts{ From efe8e6fac8bf602b3f759aa217e236701afdb7db Mon Sep 17 00:00:00 2001 From: Joseph Woodward Date: Fri, 29 Apr 2022 06:34:50 +0100 Subject: [PATCH 28/37] Document WithTimeout options for Query/QueryRange (#1037) Signed-off-by: Joseph Woodward --- api/prometheus/v1/api.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/api/prometheus/v1/api.go b/api/prometheus/v1/api.go index 83b0efe..c45e03d 100644 --- a/api/prometheus/v1/api.go +++ b/api/prometheus/v1/api.go @@ -824,6 +824,8 @@ type apiOptions struct { type Option func(c *apiOptions) +// WithTimeout can be used to provide an optional query evaluation timeout for Query and QueryRange. +// https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries func WithTimeout(timeout time.Duration) Option { return func(o *apiOptions) { o.timeout = timeout From 404809144b338c2dfdfd15a67334ad8855e24d0b Mon Sep 17 00:00:00 2001 From: S Santhosh Nagaraj Date: Fri, 29 Apr 2022 19:32:44 +0530 Subject: [PATCH 29/37] client: Allow configuration of http client (#1025) * client: Allow configuration of http client Signed-off-by: yolossn * Add api.Config validation to prevent confusion Update config documentation Signed-off-by: Kemal Akkoyun Co-authored-by: Kemal Akkoyun --- api/client.go | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/api/client.go b/api/client.go index 1413f65..c91cf0c 100644 --- a/api/client.go +++ b/api/client.go @@ -17,6 +17,7 @@ package api import ( "bytes" "context" + "errors" "net" "net/http" "net/url" @@ -40,6 +41,10 @@ type Config struct { // The address of the Prometheus to connect to. Address string + // Client is used by the Client to drive HTTP requests. If not provided, + // a new one based on the provided RoundTripper (or DefaultRoundTripper) will be used. + Client *http.Client + // RoundTripper is used by the Client to drive HTTP requests. If not // provided, DefaultRoundTripper will be used. RoundTripper http.RoundTripper @@ -52,6 +57,22 @@ func (cfg *Config) roundTripper() http.RoundTripper { return cfg.RoundTripper } +func (cfg *Config) client() http.Client { + if cfg.Client == nil { + return http.Client{ + Transport: cfg.roundTripper(), + } + } + return *cfg.Client +} + +func (cfg *Config) validate() error { + if cfg.Client != nil && cfg.RoundTripper != nil { + return errors.New("api.Config.RoundTripper and api.Config.Client are mutually exclusive") + } + return nil +} + // Client is the interface for an API client. type Client interface { URL(ep string, args map[string]string) *url.URL @@ -68,9 +89,13 @@ func NewClient(cfg Config) (Client, error) { } u.Path = strings.TrimRight(u.Path, "/") + if err := cfg.validate(); err != nil { + return nil, err + } + return &httpClient{ endpoint: u, - client: http.Client{Transport: cfg.roundTripper()}, + client: cfg.client(), }, nil } From 24605c59ac83563fc328cea83e6209ad0a01f627 Mon Sep 17 00:00:00 2001 From: Christoph Mewes Date: Mon, 2 May 2022 16:44:08 +0200 Subject: [PATCH 30/37] update branch names in a few links (#1039) Signed-off-by: Christoph Mewes --- .gitignore | 2 +- CODE_OF_CONDUCT.md | 2 +- README.md | 14 +++++++------- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.gitignore b/.gitignore index a6114ab..f080929 100644 --- a/.gitignore +++ b/.gitignore @@ -11,7 +11,7 @@ examples/random/random vendor/ # The remainder of this file is taken from -# https://github.com/github/gitignore/blob/master/Go.gitignore +# https://github.com/github/gitignore/blob/main/Go.gitignore # Binaries for programs and plugins *.exe diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 9a1aff4..07ee182 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,3 +1,3 @@ ## Prometheus Community Code of Conduct -Prometheus follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). +Prometheus follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). diff --git a/README.md b/README.md index e401975..b95a670 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Prometheus Go client library -[![CircleCI](https://circleci.com/gh/prometheus/client_golang/tree/master.svg?style=svg)](https://circleci.com/gh/prometheus/client_golang/tree/master) +[![CircleCI](https://circleci.com/gh/prometheus/client_golang/tree/main.svg?style=svg)](https://circleci.com/gh/prometheus/client_golang/tree/main) [![Go Report Card](https://goreportcard.com/badge/github.com/prometheus/client_golang)](https://goreportcard.com/report/github.com/prometheus/client_golang) [![Go Reference](https://pkg.go.dev/badge/github.com/prometheus/client_golang.svg)](https://pkg.go.dev/github.com/prometheus/client_golang) [![Slack](https://img.shields.io/badge/join%20slack-%23prometheus--client_golang-brightgreen.svg)](https://slack.cncf.io/) @@ -35,13 +35,13 @@ coexist for a while to enable a convenient transition. [![code-coverage](http://gocover.io/_badge/github.com/prometheus/client_golang/prometheus)](http://gocover.io/github.com/prometheus/client_golang/prometheus) [![Go Reference](https://pkg.go.dev/badge/github.com/prometheus/client_golang/prometheus.svg)](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus) The -[`prometheus` directory](https://github.com/prometheus/client_golang/tree/master/prometheus) +[`prometheus` directory](https://github.com/prometheus/client_golang/tree/main/prometheus) contains the instrumentation library. See the [guide](https://prometheus.io/docs/guides/go-application/) on the Prometheus website to learn more about instrumenting applications. The -[`examples` directory](https://github.com/prometheus/client_golang/tree/master/examples) +[`examples` directory](https://github.com/prometheus/client_golang/tree/main/examples) contains simple examples of instrumented code. ## Client for the Prometheus HTTP API @@ -49,7 +49,7 @@ contains simple examples of instrumented code. [![code-coverage](http://gocover.io/_badge/github.com/prometheus/client_golang/api/prometheus/v1)](http://gocover.io/github.com/prometheus/client_golang/api/prometheus/v1) [![Go Reference](https://pkg.go.dev/badge/github.com/prometheus/client_golang/api.svg)](https://pkg.go.dev/github.com/prometheus/client_golang/api) The -[`api/prometheus` directory](https://github.com/prometheus/client_golang/tree/master/api/prometheus) +[`api/prometheus` directory](https://github.com/prometheus/client_golang/tree/main/api/prometheus) contains the client for the [Prometheus HTTP API](http://prometheus.io/docs/querying/api/). It allows you to write Go applications that query time series data from a Prometheus @@ -58,14 +58,14 @@ server. It is still in alpha stage. ## Where is `model`, `extraction`, and `text`? The `model` packages has been moved to -[`prometheus/common/model`](https://github.com/prometheus/common/tree/master/model). +[`prometheus/common/model`](https://github.com/prometheus/common/tree/main/model). The `extraction` and `text` packages are now contained in -[`prometheus/common/expfmt`](https://github.com/prometheus/common/tree/master/expfmt). +[`prometheus/common/expfmt`](https://github.com/prometheus/common/tree/main/expfmt). ## Contributing and community See the [contributing guidelines](CONTRIBUTING.md) and the [Community section](http://prometheus.io/community/) of the homepage. -`clint_golang` community is also present on the CNCF Slack `#prometheus-client_golang`. +`clint_golang` community is also present on the CNCF Slack `#prometheus-client_golang`. From 0222f88f4a941059843b49d9f56b42b9fea90d97 Mon Sep 17 00:00:00 2001 From: PrometheusBot Date: Thu, 5 May 2022 07:46:54 +0200 Subject: [PATCH 31/37] Update common Prometheus files (#1045) Signed-off-by: prombot --- CODE_OF_CONDUCT.md | 2 +- SECURITY.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 07ee182..d325872 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,3 +1,3 @@ -## Prometheus Community Code of Conduct +# Prometheus Community Code of Conduct Prometheus follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). diff --git a/SECURITY.md b/SECURITY.md index 67741f0..fed02d8 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -3,4 +3,4 @@ The Prometheus security policy, including how to report vulnerabilities, can be found here: -https://prometheus.io/docs/operating/security/ + From 589b2ea560d1b157f4153f5adba5d6373cca4078 Mon Sep 17 00:00:00 2001 From: PrometheusBot Date: Fri, 6 May 2022 08:09:49 +0200 Subject: [PATCH 32/37] Update common Prometheus files (#1046) Signed-off-by: prombot --- .github/workflows/golangci-lint.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index 662ea3b..136d7d4 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -21,6 +21,9 @@ jobs: uses: actions/setup-go@v2 with: go-version: 1.18.x + - name: Install snmp_exporter/generator dependencies + run: sudo apt-get update && sudo apt-get -y install libsnmp-dev + if: github.repository == 'prometheus/snmp_exporter' - name: Lint uses: golangci/golangci-lint-action@v3.1.0 with: From f25114699aabd92765ad1ad2550a001a3b600a68 Mon Sep 17 00:00:00 2001 From: Kemal Akkoyun Date: Mon, 9 May 2022 10:33:45 +0200 Subject: [PATCH 33/37] prometheus: Fix convention violating names for generated collector metrics (#1048) * Fix convention violating names for generated collector metrics Signed-off-by: Kemal Akkoyun * Add new Go collector example Signed-off-by: Kemal Akkoyun --- .gitignore | 1 + CHANGELOG.md | 1 + Dockerfile | 7 ++- examples/gocollector/main.go | 55 +++++++++++++++++++ prometheus/collectors/go_collector_latest.go | 4 +- prometheus/gen_go_collector_metrics_set.go | 4 +- prometheus/go_collector_latest_test.go | 5 +- prometheus/go_collector_metrics_go117_test.go | 6 +- prometheus/go_collector_metrics_go118_test.go | 6 +- prometheus/internal/go_runtime_metrics.go | 14 ++--- 10 files changed, 82 insertions(+), 21 deletions(-) create mode 100644 examples/gocollector/main.go diff --git a/.gitignore b/.gitignore index f080929..788dfa1 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ # Examples examples/simple/simple examples/random/random +examples/gocollector/gocollector # Typical backup/temporary files of editors *~ diff --git a/CHANGELOG.md b/CHANGELOG.md index d515e69..7740be2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ * [CHANGE] Minimum required Go version is now 1.16. * [CHANGE] Added `collectors.WithGoCollections` that allows to choose what collection of Go runtime metrics user wants: Equivalent of [`MemStats` structure](https://pkg.go.dev/runtime#MemStats) configured using `GoRuntimeMemStatsCollection`, new based on dedicated [runtime/metrics](https://pkg.go.dev/runtime/metrics) metrics represented by `GoRuntimeMetricsCollection` option, or both by specifying `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` flag. * [CHANGE] :warning: Change in `collectors.NewGoCollector` metrics: Reverting addition of new ~80 runtime metrics by default. You can enable this back with `GoRuntimeMetricsCollection` option or `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` for smooth transition. +* [BUGFIX] Fix the bug that causes generated histogram metric names to end with `_total`. `go_gc_heap_allocs_by_size_bytes_total` -> `go_gc_heap_allocs_by_size_bytes`, `go_gc_heap_frees_by_size_bytes_total` -> `go_gc_heap_allocs_by_size_bytes` and`go_gc_pauses_seconds_total` -> `go_gc_pauses_seconds`. ## 1.12.1 / 2022-01-29 diff --git a/Dockerfile b/Dockerfile index 2627ff4..395a30a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -21,11 +21,14 @@ WORKDIR /go/src/github.com/prometheus/client_golang/examples/random RUN CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' WORKDIR /go/src/github.com/prometheus/client_golang/examples/simple RUN CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' +WORKDIR /go/src/github.com/prometheus/client_golang/examples/gocollector +RUN CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' # Final image. FROM quay.io/prometheus/busybox:latest LABEL maintainer="The Prometheus Authors " COPY --from=builder /go/src/github.com/prometheus/client_golang/examples/random \ - /go/src/github.com/prometheus/client_golang/examples/simple ./ + /go/src/github.com/prometheus/client_golang/examples/simple \ + /go/src/github.com/prometheus/client_golang/examples/gocollector ./ EXPOSE 8080 -CMD ["echo", "Please run an example. Either /random or /simple"] +CMD ["echo", "Please run an example. Either /random, /simple or /gocollector"] diff --git a/examples/gocollector/main.go b/examples/gocollector/main.go new file mode 100644 index 0000000..93f8c22 --- /dev/null +++ b/examples/gocollector/main.go @@ -0,0 +1,55 @@ +// Copyright 2022 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build go1.17 +// +build go1.17 + +// A minimal example of how to include Prometheus instrumentation. +package main + +import ( + "flag" + "fmt" + "log" + "net/http" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +var addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.") + +func main() { + flag.Parse() + + // Create a new registry. + reg := prometheus.NewRegistry() + + // Add Go module build info. + reg.MustRegister(collectors.NewBuildInfoCollector()) + reg.MustRegister(collectors.NewGoCollector( + collectors.WithGoCollections(collectors.GoRuntimeMemStatsCollection | collectors.GoRuntimeMetricsCollection), + )) + + // Expose the registered metrics via HTTP. + http.Handle("/metrics", promhttp.HandlerFor( + reg, + promhttp.HandlerOpts{ + // Opt into OpenMetrics to support exemplars. + EnableOpenMetrics: true, + }, + )) + fmt.Println("Hello world from new Go Collector!") + log.Fatal(http.ListenAndServe(*addr, nil)) +} diff --git a/prometheus/collectors/go_collector_latest.go b/prometheus/collectors/go_collector_latest.go index 58b0a5b..01790e8 100644 --- a/prometheus/collectors/go_collector_latest.go +++ b/prometheus/collectors/go_collector_latest.go @@ -72,9 +72,9 @@ const ( // // The current default is GoRuntimeMemStatsCollection, so the compatibility mode with // client_golang pre v1.12 (move to runtime/metrics). -func WithGoCollections(flags uint32) goOption { +func WithGoCollections(flags GoCollectionOption) goOption { return func(o *goOptions) { - o.EnabledCollections = flags + o.EnabledCollections = uint32(flags) } } diff --git a/prometheus/gen_go_collector_metrics_set.go b/prometheus/gen_go_collector_metrics_set.go index 4bc127c..2f60ea3 100644 --- a/prometheus/gen_go_collector_metrics_set.go +++ b/prometheus/gen_go_collector_metrics_set.go @@ -39,11 +39,11 @@ func main() { } toolVersion := runtime.Version() mtv := majorVersion(toolVersion) - mv != majorVersion(os.Args[1]) + mv := majorVersion(os.Args[1]) if mtv != mv { log.Fatalf("using Go version %q but expected Go version %q", mtv, mv) } - version, err := parseVersion(os.Args[1]) + version, err := parseVersion(mv) if err != nil { log.Fatalf("parsing Go version: %v", err) } diff --git a/prometheus/go_collector_latest_test.go b/prometheus/go_collector_latest_test.go index 88158df..11094c8 100644 --- a/prometheus/go_collector_latest_test.go +++ b/prometheus/go_collector_latest_test.go @@ -24,8 +24,9 @@ import ( "sync" "testing" - "github.com/prometheus/client_golang/prometheus/internal" dto "github.com/prometheus/client_model/go" + + "github.com/prometheus/client_golang/prometheus/internal" ) func TestRmForMemStats(t *testing.T) { @@ -121,7 +122,7 @@ func TestBatchHistogram(t *testing.T) { var mhist Metric for _, m := range goMetrics { - if m.Desc().fqName == "go_gc_heap_allocs_by_size_bytes_total" { + if m.Desc().fqName == "go_gc_heap_allocs_by_size_bytes" { mhist = m break } diff --git a/prometheus/go_collector_metrics_go117_test.go b/prometheus/go_collector_metrics_go117_test.go index 20e98ef..70c2733 100644 --- a/prometheus/go_collector_metrics_go117_test.go +++ b/prometheus/go_collector_metrics_go117_test.go @@ -10,16 +10,16 @@ var expectedRuntimeMetrics = map[string]string{ "/gc/cycles/automatic:gc-cycles": "go_gc_cycles_automatic_gc_cycles_total", "/gc/cycles/forced:gc-cycles": "go_gc_cycles_forced_gc_cycles_total", "/gc/cycles/total:gc-cycles": "go_gc_cycles_total_gc_cycles_total", - "/gc/heap/allocs-by-size:bytes": "go_gc_heap_allocs_by_size_bytes_total", + "/gc/heap/allocs-by-size:bytes": "go_gc_heap_allocs_by_size_bytes", "/gc/heap/allocs:bytes": "go_gc_heap_allocs_bytes_total", "/gc/heap/allocs:objects": "go_gc_heap_allocs_objects_total", - "/gc/heap/frees-by-size:bytes": "go_gc_heap_frees_by_size_bytes_total", + "/gc/heap/frees-by-size:bytes": "go_gc_heap_frees_by_size_bytes", "/gc/heap/frees:bytes": "go_gc_heap_frees_bytes_total", "/gc/heap/frees:objects": "go_gc_heap_frees_objects_total", "/gc/heap/goal:bytes": "go_gc_heap_goal_bytes", "/gc/heap/objects:objects": "go_gc_heap_objects_objects", "/gc/heap/tiny/allocs:objects": "go_gc_heap_tiny_allocs_objects_total", - "/gc/pauses:seconds": "go_gc_pauses_seconds_total", + "/gc/pauses:seconds": "go_gc_pauses_seconds", "/memory/classes/heap/free:bytes": "go_memory_classes_heap_free_bytes", "/memory/classes/heap/objects:bytes": "go_memory_classes_heap_objects_bytes", "/memory/classes/heap/released:bytes": "go_memory_classes_heap_released_bytes", diff --git a/prometheus/go_collector_metrics_go118_test.go b/prometheus/go_collector_metrics_go118_test.go index 2bcf545..cdef74d 100644 --- a/prometheus/go_collector_metrics_go118_test.go +++ b/prometheus/go_collector_metrics_go118_test.go @@ -10,16 +10,16 @@ var expectedRuntimeMetrics = map[string]string{ "/gc/cycles/automatic:gc-cycles": "go_gc_cycles_automatic_gc_cycles_total", "/gc/cycles/forced:gc-cycles": "go_gc_cycles_forced_gc_cycles_total", "/gc/cycles/total:gc-cycles": "go_gc_cycles_total_gc_cycles_total", - "/gc/heap/allocs-by-size:bytes": "go_gc_heap_allocs_by_size_bytes_total", + "/gc/heap/allocs-by-size:bytes": "go_gc_heap_allocs_by_size_bytes", "/gc/heap/allocs:bytes": "go_gc_heap_allocs_bytes_total", "/gc/heap/allocs:objects": "go_gc_heap_allocs_objects_total", - "/gc/heap/frees-by-size:bytes": "go_gc_heap_frees_by_size_bytes_total", + "/gc/heap/frees-by-size:bytes": "go_gc_heap_frees_by_size_bytes", "/gc/heap/frees:bytes": "go_gc_heap_frees_bytes_total", "/gc/heap/frees:objects": "go_gc_heap_frees_objects_total", "/gc/heap/goal:bytes": "go_gc_heap_goal_bytes", "/gc/heap/objects:objects": "go_gc_heap_objects_objects", "/gc/heap/tiny/allocs:objects": "go_gc_heap_tiny_allocs_objects_total", - "/gc/pauses:seconds": "go_gc_pauses_seconds_total", + "/gc/pauses:seconds": "go_gc_pauses_seconds", "/memory/classes/heap/free:bytes": "go_memory_classes_heap_free_bytes", "/memory/classes/heap/objects:bytes": "go_memory_classes_heap_objects_bytes", "/memory/classes/heap/released:bytes": "go_memory_classes_heap_released_bytes", diff --git a/prometheus/internal/go_runtime_metrics.go b/prometheus/internal/go_runtime_metrics.go index fe0a521..6cbe063 100644 --- a/prometheus/internal/go_runtime_metrics.go +++ b/prometheus/internal/go_runtime_metrics.go @@ -62,7 +62,7 @@ func RuntimeMetricsToProm(d *metrics.Description) (string, string, string, bool) // other data. name = strings.ReplaceAll(name, "-", "_") name = name + "_" + unit - if d.Cumulative { + if d.Cumulative && d.Kind != metrics.KindFloat64Histogram { name = name + "_total" } @@ -84,12 +84,12 @@ func RuntimeMetricsToProm(d *metrics.Description) (string, string, string, bool) func RuntimeMetricsBucketsForUnit(buckets []float64, unit string) []float64 { switch unit { case "bytes": - // Rebucket as powers of 2. - return rebucketExp(buckets, 2) + // Re-bucket as powers of 2. + return reBucketExp(buckets, 2) case "seconds": - // Rebucket as powers of 10 and then merge all buckets greater + // Re-bucket as powers of 10 and then merge all buckets greater // than 1 second into the +Inf bucket. - b := rebucketExp(buckets, 10) + b := reBucketExp(buckets, 10) for i := range b { if b[i] <= 1 { continue @@ -103,11 +103,11 @@ func RuntimeMetricsBucketsForUnit(buckets []float64, unit string) []float64 { return buckets } -// rebucketExp takes a list of bucket boundaries (lower bound inclusive) and +// reBucketExp takes a list of bucket boundaries (lower bound inclusive) and // downsamples the buckets to those a multiple of base apart. The end result // is a roughly exponential (in many cases, perfectly exponential) bucketing // scheme. -func rebucketExp(buckets []float64, base float64) []float64 { +func reBucketExp(buckets []float64, base float64) []float64 { bucket := buckets[0] var newBuckets []float64 // We may see a -Inf here, in which case, add it and skip it From 35c82f2c7ee8a3f535cf3494cd4b2d9b4cba60b1 Mon Sep 17 00:00:00 2001 From: Kemal Akkoyun Date: Fri, 13 May 2022 10:04:45 +0200 Subject: [PATCH 34/37] Remove -Inf buckets from go collector histograms (#1049) * Remove -Inf buckets from go collector histograms Signed-off-by: Kemal Akkoyun * Update prometheus/collectors/go_collector_latest_test.go Co-authored-by: Bartlomiej Plotka Signed-off-by: Kemal Akkoyun * Simplify Signed-off-by: Kemal Akkoyun Co-authored-by: Bartlomiej Plotka --- .../collectors/go_collector_latest_test.go | 39 +++++++++++++++++++ prometheus/go_collector_latest.go | 14 +++++-- prometheus/go_collector_metrics_go117_test.go | 2 +- prometheus/go_collector_metrics_go118_test.go | 2 +- 4 files changed, 52 insertions(+), 5 deletions(-) create mode 100644 prometheus/collectors/go_collector_latest_test.go diff --git a/prometheus/collectors/go_collector_latest_test.go b/prometheus/collectors/go_collector_latest_test.go new file mode 100644 index 0000000..126864c --- /dev/null +++ b/prometheus/collectors/go_collector_latest_test.go @@ -0,0 +1,39 @@ +// Copyright 2021 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build go1.17 +// +build go1.17 + +package collectors + +import ( + "encoding/json" + "testing" + + "github.com/prometheus/client_golang/prometheus" +) + +func TestGoCollectorMarshalling(t *testing.T) { + reg := prometheus.NewRegistry() + reg.MustRegister(NewGoCollector( + WithGoCollections(GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection), + )) + result, err := reg.Gather() + if err != nil { + t.Fatal(err) + } + + if _, err := json.Marshal(result); err != nil { + t.Errorf("json marshalling shoud not fail, %v", err) + } +} diff --git a/prometheus/go_collector_latest.go b/prometheus/go_collector_latest.go index 8528ea7..a0fe95e 100644 --- a/prometheus/go_collector_latest.go +++ b/prometheus/go_collector_latest.go @@ -25,8 +25,9 @@ import ( //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility. "github.com/golang/protobuf/proto" - "github.com/prometheus/client_golang/prometheus/internal" dto "github.com/prometheus/client_model/go" + + "github.com/prometheus/client_golang/prometheus/internal" ) const ( @@ -429,6 +430,11 @@ type batchHistogram struct { // buckets must always be from the runtime/metrics package, following // the same conventions. func newBatchHistogram(desc *Desc, buckets []float64, hasSum bool) *batchHistogram { + // We need to remove -Inf values. runtime/metrics keeps them around. + // But -Inf bucket should not be allowed for prometheus histograms. + if buckets[0] == math.Inf(-1) { + buckets = buckets[1:] + } h := &batchHistogram{ desc: desc, buckets: buckets, @@ -487,8 +493,10 @@ func (h *batchHistogram) Write(out *dto.Metric) error { for i, count := range h.counts { totalCount += count if !h.hasSum { - // N.B. This computed sum is an underestimate. - sum += h.buckets[i] * float64(count) + if count != 0 { + // N.B. This computed sum is an underestimate. + sum += h.buckets[i] * float64(count) + } } // Skip the +Inf bucket, but only for the bucket list. diff --git a/prometheus/go_collector_metrics_go117_test.go b/prometheus/go_collector_metrics_go117_test.go index 70c2733..1b8a869 100644 --- a/prometheus/go_collector_metrics_go117_test.go +++ b/prometheus/go_collector_metrics_go117_test.go @@ -38,4 +38,4 @@ var expectedRuntimeMetrics = map[string]string{ "/sched/latencies:seconds": "go_sched_latencies_seconds", } -const expectedRuntimeMetricsCardinality = 79 +const expectedRuntimeMetricsCardinality = 77 diff --git a/prometheus/go_collector_metrics_go118_test.go b/prometheus/go_collector_metrics_go118_test.go index cdef74d..44bfb86 100644 --- a/prometheus/go_collector_metrics_go118_test.go +++ b/prometheus/go_collector_metrics_go118_test.go @@ -38,4 +38,4 @@ var expectedRuntimeMetrics = map[string]string{ "/sched/latencies:seconds": "go_sched_latencies_seconds", } -const expectedRuntimeMetricsCardinality = 79 +const expectedRuntimeMetricsCardinality = 77 From edecbb214b8a8bb1380b2ac10241736ad8824589 Mon Sep 17 00:00:00 2001 From: Ben Kochie Date: Fri, 13 May 2022 10:05:17 +0200 Subject: [PATCH 35/37] Enable dependabot (#1050) Update Go deps monthly. Signed-off-by: SuperQ --- .github/dependabot.yml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..202ae23 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,6 @@ +version: 2 +updates: + - package-ecosystem: "gomod" + directory: "/" + schedule: + interval: "monthly" From fab6748c348fc7c09647b1b1525460e4e0a4681a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 13 May 2022 10:20:41 +0200 Subject: [PATCH 36/37] Bump github.com/prometheus/common from 0.33.0 to 0.34.0 (#1051) Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.33.0 to 0.34.0. - [Release notes](https://github.com/prometheus/common/releases) - [Commits](https://github.com/prometheus/common/compare/v0.33.0...v0.34.0) --- updated-dependencies: - dependency-name: github.com/prometheus/common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 5135421..1105fe8 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/jpillora/backoff v1.0.0 // indirect github.com/json-iterator/go v1.1.12 github.com/prometheus/client_model v0.2.0 - github.com/prometheus/common v0.33.0 + github.com/prometheus/common v0.34.0 github.com/prometheus/procfs v0.7.3 golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886 google.golang.org/protobuf v1.28.0 diff --git a/go.sum b/go.sum index 2f3a451..379f2c8 100644 --- a/go.sum +++ b/go.sum @@ -136,8 +136,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M= github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/common v0.33.0 h1:rHgav/0a6+uYgGdNt3jwz8FNSesO/Hsang3O0T9A5SE= -github.com/prometheus/common v0.33.0/go.mod h1:gB3sOl7P0TvJabZpLY5uQMpUqRCPPCyRLCZYc7JZTNE= +github.com/prometheus/common v0.34.0 h1:RBmGO9d/FVjqHT0yUGQwBJhkwKV+wPCn7KGpvfab0uE= +github.com/prometheus/common v0.34.0/go.mod h1:gB3sOl7P0TvJabZpLY5uQMpUqRCPPCyRLCZYc7JZTNE= github.com/prometheus/procfs v0.7.3 h1:4jVXhlkAyzOScmCkXBTOLRLTz8EeU+eyjrwB/EPq0VU= github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= From 5d584e2717ef525673736d72cd1d12e304f243d7 Mon Sep 17 00:00:00 2001 From: Kemal Akkoyun Date: Sat, 14 May 2022 10:10:15 +0200 Subject: [PATCH 37/37] Update readme (#1053) Signed-off-by: Kemal Akkoyun --- CHANGELOG.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7740be2..504f0fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,16 @@ ## Unreleased * [CHANGE] Minimum required Go version is now 1.16. + +## 1.12.2 / 2022-05-13 + * [CHANGE] Added `collectors.WithGoCollections` that allows to choose what collection of Go runtime metrics user wants: Equivalent of [`MemStats` structure](https://pkg.go.dev/runtime#MemStats) configured using `GoRuntimeMemStatsCollection`, new based on dedicated [runtime/metrics](https://pkg.go.dev/runtime/metrics) metrics represented by `GoRuntimeMetricsCollection` option, or both by specifying `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` flag. -* [CHANGE] :warning: Change in `collectors.NewGoCollector` metrics: Reverting addition of new ~80 runtime metrics by default. You can enable this back with `GoRuntimeMetricsCollection` option or `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` for smooth transition. -* [BUGFIX] Fix the bug that causes generated histogram metric names to end with `_total`. `go_gc_heap_allocs_by_size_bytes_total` -> `go_gc_heap_allocs_by_size_bytes`, `go_gc_heap_frees_by_size_bytes_total` -> `go_gc_heap_allocs_by_size_bytes` and`go_gc_pauses_seconds_total` -> `go_gc_pauses_seconds`. +* [CHANGE] :warning: Change in `collectors.NewGoCollector` metrics: Reverting addition of new ~80 runtime metrics by default. You can enable this back with `GoRuntimeMetricsCollection` option or `GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection` for smooth transition. +* [BUGFIX] Fixed the bug that causes generated histogram metric names to end with `_total`. ⚠️ This changes 3 metric names in the new Go collector that was reverted from default in this release. + * `go_gc_heap_allocs_by_size_bytes_total` -> `go_gc_heap_allocs_by_size_bytes`, + * `go_gc_heap_frees_by_size_bytes_total` -> `go_gc_heap_allocs_by_size_bytes` + * `go_gc_pauses_seconds_total` -> `go_gc_pauses_seconds`. +* [CHANCE] Removed `-Inf` buckets from new Go Collector histograms. ## 1.12.1 / 2022-01-29