Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services/horizon: Allow starting Horizon with minimal configuration #3783

Merged
merged 7 commits into from
Jul 30, 2021
Merged

services/horizon: Allow starting Horizon with minimal configuration #3783

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d71d976
services/horizon: Allow starting Horizon with minimal configuration
bartekn Jul 26, 2021
246ddbd
pubnet
bartekn Jul 27, 2021
8b85c49
Review feedback
bartekn Jul 30, 2021
3a9c69c
Merge branch 'master' into horizon-minimal-config
bartekn Jul 30, 2021
e1c4cc6
check if file exist
bartekn Jul 30, 2021
0ce7936
Merge branch 'horizon-minimal-config' of github.com:bartekn/go into h…
bartekn Jul 30, 2021
055deda
fix shadow
bartekn Jul 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
259 changes: 259 additions & 0 deletions ingest/ledgerbackend/toml.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"regexp"
"strings"

"github.com/stellar/go/network"
"github.com/stellar/go/support/errors"
"github.com/stellar/go/support/log"
"github.com/stellar/go/xdr"
Expand Down Expand Up @@ -348,6 +349,264 @@ func NewCaptiveCoreTomlFromFile(configPath string, params CaptiveCoreTomlParams)
return &captiveCoreToml, nil
}

// NewDefaultTestnetCaptiveCoreToml constructs a new CaptiveCoreToml instance
// based off the default testnet configuration.
func NewDefaultTestnetCaptiveCoreToml() *CaptiveCoreToml {
var captiveCoreToml CaptiveCoreToml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be easier to maintain these defaults if we used raw strings?

For example, we could use this config for the testnet and just write it to / read it from an ioutil.TempFile. I think it'd be easier to maintain if the configs changed over time, since you could just re-copy-paste the config instead of trying to identify the delta and adapting it to our internal data structure.

Copy link
Contributor

@tamirms tamirms Jul 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have the default pubnet and testnet validators represented in string form. Then we can implement NewDefaultTestnetCaptiveCoreToml() and NewDefaultPubnetCaptiveCoreToml() by calling newCaptiveCoreTomlFromFile(tomlFileContents string, params CaptiveCoreTomlParams) (*CaptiveCoreToml, error)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After @jacekn comment and given that this feature will be used probably only by users who use GH release maybe instead of keeping the file in the binary we could add testnet and pubnet config files in the packed .tar.gz release file. Then we can change the code to try to load the default file from disk if the value is not explicitly set.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that idea

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the file idea to the curl idea because curl'd endpoints can change or go away (or be compromised), so less reproducible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the file idea to the curl idea because curl'd endpoints can change or go away (or be compromised), so less reproducible.

If you use curl it would be best to put the files in a location controlled by SDF, possibly in the go repo next to horizon source code. That way security profile won't change, we'd essentially get the same level of security as for the horizon source code itself.

Copy link
Contributor Author

@bartekn bartekn Jul 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just pushed a change adding two default config files to release archive here: 8b85c49.


captiveCoreToml.tablePlaceholders = &placeholders{}

captiveCoreToml.PublicHTTPPort = true
captiveCoreToml.HTTPPort = 11626

captiveCoreToml.FailureSafety = -1
captiveCoreToml.NetworkPassphrase = network.TestNetworkPassphrase

captiveCoreToml.HomeDomains = []HomeDomain{
{
HomeDomain: "testnet.stellar.org",
Quality: "LOW",
},
}

captiveCoreToml.Validators = []Validator{
{
Name: "sdf_testnet_1",
HomeDomain: "testnet.stellar.org",
PublicKey: "GDKXE2OZMJIPOSLNA6N6F2BVCI3O777I2OOC4BV7VOYUEHYX7RTRYA7Y",
Address: "core-testnet1.stellar.org",
History: "curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_001/{0} -o {1}",
},
{
Name: "sdf_testnet_2",
HomeDomain: "testnet.stellar.org",
PublicKey: "GCUCJTIYXSOXKBSNFGNFWW5MUQ54HKRPGJUTQFJ5RQXZXNOLNXYDHRAP",
Address: "core-testnet2.stellar.org",
History: "curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_002/{0} -o {1}",
},
{
Name: "sdf_testnet_3",
HomeDomain: "testnet.stellar.org",
PublicKey: "GC2V2EFSXN6SQTWVYA5EPJPBWWIMSD2XQNKUOHGEKB535AQE2I6IXV2Z",
Address: "core-testnet3.stellar.org",
History: "curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_003/{0} -o {1}",
},
}

return &captiveCoreToml
}

// NewDefaultPubnetCaptiveCoreToml constructs a new CaptiveCoreToml instance
// based off the default pubnet configuration.
func NewDefaultPubnetCaptiveCoreToml() *CaptiveCoreToml {
var captiveCoreToml CaptiveCoreToml

captiveCoreToml.tablePlaceholders = &placeholders{}

captiveCoreToml.PublicHTTPPort = true
captiveCoreToml.HTTPPort = 11626

captiveCoreToml.FailureSafety = -1
captiveCoreToml.NetworkPassphrase = network.PublicNetworkPassphrase

captiveCoreToml.HomeDomains = []HomeDomain{
{
HomeDomain: "stellar.org",
Quality: "HIGH",
},
{
HomeDomain: "satoshipay.io",
Quality: "HIGH",
},
{
HomeDomain: "lobstr.co",
Quality: "HIGH",
},
{
HomeDomain: "www.coinqvest.com",
Quality: "HIGH",
},
{
HomeDomain: "keybase.io",
Quality: "HIGH",
},
{
HomeDomain: "stellar.blockdaemon.com",
Quality: "HIGH",
},
{
HomeDomain: "wirexapp.com",
Quality: "HIGH",
},
}

captiveCoreToml.Validators = []Validator{
{
Name: "sdf_1",
HomeDomain: "stellar.org",
PublicKey: "GCGB2S2KGYARPVIA37HYZXVRM2YZUEXA6S33ZU5BUDC6THSB62LZSTYH",
Address: "core-live-a.stellar.org:11625",
History: "curl -sf https://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}",
},
{
Name: "sdf_2",
HomeDomain: "stellar.org",
PublicKey: "GCM6QMP3DLRPTAZW2UZPCPX2LF3SXWXKPMP3GKFZBDSF3QZGV2G5QSTK",
Address: "core-live-b.stellar.org:11625",
History: "curl -sf https://history.stellar.org/prd/core-live/core_live_002/{0} -o {1}",
},
{
Name: "sdf_3",
HomeDomain: "stellar.org",
PublicKey: "GABMKJM6I25XI4K7U6XWMULOUQIQ27BCTMLS6BYYSOWKTBUXVRJSXHYQ",
Address: "core-live-c.stellar.org:11625",
History: "curl -sf https://history.stellar.org/prd/core-live/core_live_003/{0} -o {1}",
},
{
Name: "satoshipay_singapore",
HomeDomain: "satoshipay.io",
PublicKey: "GBJQUIXUO4XSNPAUT6ODLZUJRV2NPXYASKUBY4G5MYP3M47PCVI55MNT",
Address: "stellar-sg-sin.satoshipay.io:11625",
History: "curl -sf https://stellar-history-sg-sin.satoshipay.io/{0} -o {1}",
},
{
Name: "satoshipay_iowa",
HomeDomain: "satoshipay.io",
PublicKey: "GAK6Z5UVGUVSEK6PEOCAYJISTT5EJBB34PN3NOLEQG2SUKXRVV2F6HZY",
Address: "stellar-us-iowa.satoshipay.io:11625",
History: "curl -sf https://stellar-history-us-iowa.satoshipay.io/{0} -o {1}",
},
{
Name: "satoshipay_frankfurt",
HomeDomain: "satoshipay.io",
PublicKey: "GC5SXLNAM3C4NMGK2PXK4R34B5GNZ47FYQ24ZIBFDFOCU6D4KBN4POAE",
Address: "stellar-de-fra.satoshipay.io:11625",
History: "curl -sf https://stellar-history-de-fra.satoshipay.io/{0} -o {1}",
},
{
Name: "lobstr_1_europe",
HomeDomain: "lobstr.co",
PublicKey: "GCFONE23AB7Y6C5YZOMKUKGETPIAJA4QOYLS5VNS4JHBGKRZCPYHDLW7",
Address: "v1.stellar.lobstr.co:11625",
History: "curl -sf https://stellar-archive-1-lobstr.s3.amazonaws.com/{0} -o {1}",
},
{
Name: "lobstr_2_europe",
HomeDomain: "lobstr.co",
PublicKey: "GDXQB3OMMQ6MGG43PWFBZWBFKBBDUZIVSUDAZZTRAWQZKES2CDSE5HKJ",
Address: "v2.stellar.lobstr.co:11625",
History: "curl -sf https://stellar-archive-2-lobstr.s3.amazonaws.com/{0} -o {1}",
},
{
Name: "lobstr_3_north_america",
HomeDomain: "lobstr.co",
PublicKey: "GD5QWEVV4GZZTQP46BRXV5CUMMMLP4JTGFD7FWYJJWRL54CELY6JGQ63",
Address: "v3.stellar.lobstr.co:11625",
History: "curl -sf https://stellar-archive-3-lobstr.s3.amazonaws.com/{0} -o {1}",
},
{
Name: "lobstr_4_asia",
HomeDomain: "lobstr.co",
PublicKey: "GA7TEPCBDQKI7JQLQ34ZURRMK44DVYCIGVXQQWNSWAEQR6KB4FMCBT7J",
Address: "v4.stellar.lobstr.co:11625",
History: "curl -sf https://stellar-archive-4-lobstr.s3.amazonaws.com/{0} -o {1}",
},
{
Name: "lobstr_5_australia",
HomeDomain: "lobstr.co",
PublicKey: "GA5STBMV6QDXFDGD62MEHLLHZTPDI77U3PFOD2SELU5RJDHQWBR5NNK7",
Address: "v5.stellar.lobstr.co:11625",
History: "curl -sf https://stellar-archive-5-lobstr.s3.amazonaws.com/{0} -o {1}",
},
{
Name: "coinqvest_hong_kong",
HomeDomain: "www.coinqvest.com",
PublicKey: "GAZ437J46SCFPZEDLVGDMKZPLFO77XJ4QVAURSJVRZK2T5S7XUFHXI2Z",
Address: "hongkong.stellar.coinqvest.com:11625",
History: "curl -sf https://hongkong.stellar.coinqvest.com/history/{0} -o {1}",
},
{
Name: "coinqvest_germany",
HomeDomain: "www.coinqvest.com",
PublicKey: "GD6SZQV3WEJUH352NTVLKEV2JM2RH266VPEM7EH5QLLI7ZZAALMLNUVN",
Address: "germany.stellar.coinqvest.com:11625",
History: "curl -sf https://germany.stellar.coinqvest.com/history/{0} -o {1}",
},
{
Name: "coinqvest_finland",
HomeDomain: "www.coinqvest.com",
PublicKey: "GADLA6BJK6VK33EM2IDQM37L5KGVCY5MSHSHVJA4SCNGNUIEOTCR6J5T",
Address: "finland.stellar.coinqvest.com:11625",
History: "curl -sf https://finland.stellar.coinqvest.com/history/{0} -o {1}",
},
{
Name: "keybase_io",
HomeDomain: "keybase.io",
PublicKey: "GCWJKM4EGTGJUVSWUJDPCQEOEP5LHSOFKSA4HALBTOO4T4H3HCHOM6UX",
Address: "stellar0.keybase.io:11625",
History: "curl -sf https://stellarhistory.keybase.io/{0} -o {1}",
},
{
Name: "keybase_1",
HomeDomain: "keybase.io",
PublicKey: "GDKWELGJURRKXECG3HHFHXMRX64YWQPUHKCVRESOX3E5PM6DM4YXLZJM",
Address: "stellar1.keybase.io:11625",
History: "curl -sf https://stellarhistory1.keybase.io/{0} -o {1}",
},
{
Name: "keybase_2",
HomeDomain: "keybase.io",
PublicKey: "GA35T3723UP2XJLC2H7MNL6VMKZZIFL2VW7XHMFFJKKIA2FJCYTLKFBW",
Address: "stellar2.keybase.io:11625",
History: "curl -sf https://stellarhistory2.keybase.io/{0} -o {1}",
},
{
Name: "Blockdaemon_Validator_1",
HomeDomain: "stellar.blockdaemon.com",
PublicKey: "GAAV2GCVFLNN522ORUYFV33E76VPC22E72S75AQ6MBR5V45Z5DWVPWEU",
Address: "stellar-full-validator1.bdnodes.net",
History: "curl -sf https://stellar-full-history1.bdnodes.net/{0} -o {1}",
},
{
Name: "Blockdaemon_Validator_2",
HomeDomain: "stellar.blockdaemon.com",
PublicKey: "GAVXB7SBJRYHSG6KSQHY74N7JAFRL4PFVZCNWW2ARI6ZEKNBJSMSKW7C",
Address: "stellar-full-validator2.bdnodes.net",
History: "curl -sf https://stellar-full-history2.bdnodes.net/{0} -o {1}",
},
{
Name: "Blockdaemon_Validator_3",
HomeDomain: "stellar.blockdaemon.com",
PublicKey: "GAYXZ4PZ7P6QOX7EBHPIZXNWY4KCOBYWJCA4WKWRKC7XIUS3UJPT6EZ4",
Address: "stellar-full-validator3.bdnodes.net",
History: "curl -sf https://stellar-full-history3.bdnodes.net/{0} -o {1}",
},
{
Name: "wirexUS",
HomeDomain: "wirexapp.com",
PublicKey: "GDXUKFGG76WJC7ACEH3JUPLKM5N5S76QSMNDBONREUXPCZYVPOLFWXUS",
Address: "us.stellar.wirexapp.com",
History: "curl -sf http://wxhorizonusstga1.blob.core.windows.net/history/{0} -o {1}",
},
{
Name: "wirexUK",
HomeDomain: "wirexapp.com",
PublicKey: "GBBQQT3EIUSXRJC6TGUCGVA3FVPXVZLGG3OJYACWBEWYBHU46WJLWXEU",
Address: "uk.stellar.wirexapp.com",
History: "curl -sf http://wxhorizonukstga1.blob.core.windows.net/history/{0} -o {1}",
},
{
Name: "wirexSG",
HomeDomain: "wirexapp.com",
PublicKey: "GAB3GZIE6XAYWXGZUDM4GMFFLJBFMLE2JDPUCWUZXMOMT3NHXDHEWXAS",
Address: "sg.stellar.wirexapp.com",
History: "curl -sf http://wxhorizonasiastga1.blob.core.windows.net/history/{0} -o {1}",
},
}

return &captiveCoreToml
}

// NewCaptiveCoreToml constructs a new CaptiveCoreToml instance based off
// the configuration in `params`.
func NewCaptiveCoreToml(params CaptiveCoreTomlParams) (*CaptiveCoreToml, error) {
Expand Down
1 change: 1 addition & 0 deletions services/horizon/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ file. This project adheres to [Semantic Versioning](http://semver.org/).

## Unreleased

* The `--ingest` flag is set by default. If `--captive-core-config-path` is not set, the config file is generated based on network passhprase. ([3783](https://github.com/stellar/go/pull/3783))
* Add a feature flag `--captive-core-reuse-storage-path`/`CAPTIVE_CORE_REUSE_STORAGE_PATH` that will reuse Captive Core's storage path for bucket files when applicable for better performance ([3750](https://github.com/stellar/go/pull/3750)).

## v2.6.1
Expand Down
15 changes: 15 additions & 0 deletions services/horizon/internal/app.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,21 @@ func (a *App) Serve() {
// configure shutdown signal handler
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)

if a.config.UsingDefaultPubnetConfig {
bartekn marked this conversation as resolved.
Show resolved Hide resolved
// If Horizon was started using default pubnet configuration kill it
// after one hour with error message.
const errorMsg = "Horizon started using the default pubnet configuration. " +
"Because this is not safe it will be shut down after one hour. " +
"Please provide a custom --captive-core-config-path."
log.Error(errorMsg)
go func() {
time.Sleep(time.Hour)
log.Error(errorMsg)
signalChan <- syscall.SIGINT
}()
}

go func() {
select {
case <-signalChan:
Expand Down
1 change: 1 addition & 0 deletions services/horizon/internal/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ type Config struct {
AdminPort uint

EnableCaptiveCoreIngestion bool
UsingDefaultPubnetConfig bool
CaptiveCoreBinaryPath string
RemoteCaptiveCoreURL string
CaptiveCoreConfigPath string
Expand Down
17 changes: 14 additions & 3 deletions services/horizon/internal/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"github.com/spf13/viper"

"github.com/stellar/go/ingest/ledgerbackend"
"github.com/stellar/go/network"
"github.com/stellar/go/services/horizon/internal/db2/schema"
apkg "github.com/stellar/go/support/app"
support "github.com/stellar/go/support/config"
Expand Down Expand Up @@ -389,7 +390,7 @@ func Flags() (*Config, support.ConfigOptions) {
Name: "ingest",
ConfigKey: &config.Ingest,
OptType: types.Bool,
FlagDefault: false,
FlagDefault: true,
Usage: "causes this horizon process to ingest data from stellar-core into horizon's db",
},
&support.ConfigOption{
Expand Down Expand Up @@ -538,8 +539,18 @@ func ApplyFlags(config *Config, flags support.ConfigOptions, options ApplyOption

if config.RemoteCaptiveCoreURL == "" && (binaryPath == "" || config.CaptiveCoreConfigPath == "") {
if options.RequireCaptiveCoreConfig {
stdLog.Fatalf("Invalid config: captive core requires that both --%s and --%s are set. %s",
StellarCoreBinaryPathName, CaptiveCoreConfigPathName, captiveCoreMigrationHint)
switch config.NetworkPassphrase {
case network.TestNetworkPassphrase:
bartekn marked this conversation as resolved.
Show resolved Hide resolved
config.CaptiveCoreToml = ledgerbackend.NewDefaultTestnetCaptiveCoreToml()
config.HistoryArchiveURLs = []string{"https://history.stellar.org/prd/core-testnet/core_testnet_001/"}
case network.PublicNetworkPassphrase:
config.CaptiveCoreToml = ledgerbackend.NewDefaultPubnetCaptiveCoreToml()
config.HistoryArchiveURLs = []string{"https://history.stellar.org/prd/core-live/core_live_001/"}
config.UsingDefaultPubnetConfig = true
default:
stdLog.Fatalf("Invalid config: captive core requires that both --%s and --%s are set. %s",
StellarCoreBinaryPathName, CaptiveCoreConfigPathName, captiveCoreMigrationHint)
}
} else {
var err error
config.CaptiveCoreTomlParams.HistoryArchiveURLs = config.HistoryArchiveURLs
Expand Down