Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: trending queries #301

Merged
merged 40 commits into from
Nov 21, 2024
Merged

feat: trending queries #301

Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
d7bdf88
chore: installs sdk 0.2.5 locally, preps branch
grantdfoster Oct 31, 2024
ae05b2d
feat: incorporates new sdk, removes count and twitter config, updates…
grantdfoster Oct 31, 2024
d17dd34
fix: timedelta
grantdfoster Oct 31, 2024
89c5e39
fix: validation function name
grantdfoster Oct 31, 2024
cbcf1a8
feat: miners define max count
grantdfoster Oct 31, 2024
69b8853
fix: cleans up timeout and counts
grantdfoster Oct 31, 2024
8774387
feat: reduce randomness (#302)
grantdfoster Nov 1, 2024
0bf8f15
fix: increased test coverage
grantdfoster Nov 3, 2024
d917ff5
fix: cleans up tests
grantdfoster Nov 5, 2024
1ef9dca
fix: makefile
grantdfoster Nov 5, 2024
79a604f
fix: pushing config
grantdfoster Nov 5, 2024
7bfa485
fix: backup fetch
grantdfoster Nov 5, 2024
780fcd7
fix: naming
grantdfoster Nov 5, 2024
9f923a7
fix: config and mapping
grantdfoster Nov 5, 2024
37933ee
fix: removes mock
grantdfoster Nov 5, 2024
d7c68d1
fix: config for network
grantdfoster Nov 5, 2024
f069a95
chore: ports entire config
grantdfoster Nov 5, 2024
9c88bc1
fix: adds tests
grantdfoster Nov 5, 2024
1792cb3
fix: reverts makefile
grantdfoster Nov 5, 2024
4f5ad44
fix: refactor forwarder
grantdfoster Nov 5, 2024
9745fb8
fix: since
grantdfoster Nov 5, 2024
1a98740
fix: update config
grantdfoster Nov 5, 2024
afafb01
fix: tempo blocktime
grantdfoster Nov 5, 2024
d5b003e
fix: test error string
grantdfoster Nov 5, 2024
efc3679
fix: bump version
grantdfoster Nov 5, 2024
3e27141
fix: more tests
grantdfoster Nov 5, 2024
8be84a8
fix: scoring test
grantdfoster Nov 5, 2024
0b1f1b9
fix: cleanup unused files like docker and scrips
grantdfoster Nov 6, 2024
38c23da
fix: remove miner test for protocol
grantdfoster Nov 6, 2024
e4bffa4
feat: adds max tweet count and refactors synapses
grantdfoster Nov 6, 2024
abe2118
fix: healthcheck import
grantdfoster Nov 6, 2024
3aae362
fix: import
grantdfoster Nov 6, 2024
d35e655
fix: reverts makefile for deploy
grantdfoster Nov 6, 2024
f970b8b
fix: Fix tests by adding registered miner and validator wallets as se…
5u6r054 Nov 7, 2024
d7371f7
Merge branch 'main' into feat--trending-queries
grantdfoster Nov 7, 2024
470887d
fix: vali test
grantdfoster Nov 7, 2024
bfcf40a
fix: twitter limit and makefile logging
grantdfoster Nov 19, 2024
161e0bb
fix: state loading guards
grantdfoster Nov 19, 2024
0a626d6
fix: improved error handling in recent tweets
grantdfoster Nov 20, 2024
14fa81e
fix: adds requests library for better error handling
grantdfoster Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
VALIDATOR_API_HOST=localhost
VALIDATOR_API_HOST=127.0.0.1
VALIDATOR_API_PORT=8000
ORACLE_BASE_URL=http://localhost:8080/api/v1
ORACLE_BASE_URL=http://127.0.0.1:8080/api/v1
129 changes: 10 additions & 119 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,47 +1,20 @@
BRANCH_NAME := $(shell git rev-parse --abbrev-ref HEAD)
DOCKER_COMPOSE := BRANCH_NAME=$(BRANCH_NAME) docker compose

DEV_NET_ENDPOINT = ws://100.28.51.29:9945
DEV_NET = chain_endpoint $(DEV_NET_ENDPOINT)

TESTNET = network test
MAINNET = network finney

########################################################################
##### SELECT YOUR ENV #####
########################################################################

# SUBTENSOR_ENVIRONMENT = $(DEV_NET)
# SUBTENSOR_ENVIRONMENT = $(TESTNET)
SUBTENSOR_ENVIRONMENT = $(MAINNET)

# NETUID = 1 # devnet
# NETUID = 165 # testnet
NETUID = 42 # mainnet

########################################################################
##### USEFUL COMMANDS #####
########################################################################

## Wallet funding
fund-owner-wallet:
btcli wallet faucet --wallet.name owner --subtensor.$(SUBTENSOR_ENVIRONMENT)

fund-validator-wallet:
btcli wallet faucet --wallet.name validator --subtensor.$(SUBTENSOR_ENVIRONMENT)

fund-miner-wallet:
btcli wallet faucet --wallet.name miner --subtensor.$(SUBTENSOR_ENVIRONMENT)

## Send TAO
send:
btcli w transfer --subtensor.$(SUBTENSOR_ENVIRONMENT)

## Subnet creation
create-subnet:
btcli subnet create --wallet.name owner --subtensor.$(SUBTENSOR_ENVIRONMENT)

## Subnet and wallet info
list-wallets:
btcli wallet list

Expand All @@ -54,123 +27,41 @@ balance-all:
list-subnets:
btcli subnets list --subtensor.$(SUBTENSOR_ENVIRONMENT)

## Validator setup
stake-validator:
btcli stake add --wallet.name validator --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)
register-miner:
btcli subnet register --wallet.name miner --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)

register-validator:
btcli subnet register --wallet.name validator --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)

register-validator-root:
btcli root register --wallet.name validator --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT)

## Register miner + Key Registration Validation
register-miner:
btcli subnet register --wallet.name miner --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)

validate-key-registration:
btcli subnet list --subtensor.$(SUBTENSOR_ENVIRONMENT)
stake-validator:
btcli stake add --wallet.name validator --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)

## Setup weights
boost-root:
btcli root boost --netuid $(NETUID) --increase 1 --wallet.name validator --wallet.hotkey default --subtensor.$(SUBTENSOR_ENVIRONMENT)

set-weights:
btcli root weights --subtensor.$(SUBTENSOR_ENVIRONMENT)

## Run miner and validator
run-miner:
python neurons/miner.py --netuid $(NETUID) --subtensor.$(SUBTENSOR_ENVIRONMENT) --wallet.name miner --wallet.hotkey default --axon.port 8091 --neuron.debug --logging.debug --blacklist.force_validator_permit
python neurons/miner.py --netuid $(NETUID) --subtensor.$(SUBTENSOR_ENVIRONMENT) --wallet.name miner --wallet.hotkey default --axon.port 8091 --neuron.debug --logging.debug --logging.logging_dir ~/.bittensor/wallets --blacklist.force_validator_permit

run-validator:
python neurons/validator.py --netuid $(NETUID) --subtensor.$(SUBTENSOR_ENVIRONMENT) --wallet.name validator --wallet.hotkey default --axon.port 8092 --neuron.debug --logging.debug --neuron.axon_off

## Docker commands
docker-build:
docker build -f Dockerfile.masa -t masa-subtensor .

docker-run:
docker run -d --name masa-subtensor -p 30333:30333 -p 9933:9933 -p 9944:9944 -p 9945:9945 -p 9946:9946 masa-subtensor

docker-run-remote:
docker run -d --name masa-subtensor -p 30333:30333 -p 9933:9933 -p 9944:9944 -p 9945:9945 -p 9946:9946 ghcr.io/masa-finance/subtensor:arm-latest


########################################################################
##### VALIDATOR API #####
########################################################################

test-profile:
curl -X GET "http://localhost:8000/data/twitter/brendanplayford" -H "Authorization: Bearer 1234"
python neurons/validator.py --netuid $(NETUID) --subtensor.$(SUBTENSOR_ENVIRONMENT) --wallet.name validator --wallet.hotkey default --axon.port 8092 --neuron.debug --logging.debug --logging.logging_dir ~/.bittensor/wallets --neuron.axon_off

########################################################################
########################################################################


## Helpful commands (generally don't run well on their own with Makefiles)
## because each line in recipe runs on its own shell invocation
activate-environment:
conda activate bittensor
bittensor-path-export:
export PYTHONPATH=$(BITTENSOR_PATH)
########################################################################

## Subtensor repo related commands (not needed here)
build-binary:
cargo build --release --features pow-faucet

run-localnet:
BUILD_BINARY=0 ./scripts/localnet.sh

## Hyperparameters and metagraph
hyperparameters:
btcli subnets hyperparameters --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)

metagraph:
btcli subnets metagraph --subtensor.$(SUBTENSOR_ENVIRONMENT) --netuid $(NETUID)

set-hyperparam:
btcli sudo set --param weights_version --value 10 --netuid $(NETUID) --subtensor.$(SUBTENSOR_ENVIRONMENT)

########################################################################
##### DOCKER COMPOSE COMMANDS #####
########################################################################

.PHONY: up down build logs

pull:
$(DOCKER_COMPOSE) pull

up:
$(DOCKER_COMPOSE) up -d --pull always

down:
$(DOCKER_COMPOSE) down

build:
$(DOCKER_COMPOSE) build

logs:
$(DOCKER_COMPOSE) logs -f

# You can keep your existing docker commands or replace them with these:
docker-build:
$(DOCKER_COMPOSE) build

docker-up:
$(DOCKER_COMPOSE) up -d --pull always

docker-down:
$(DOCKER_COMPOSE) down

test-mocks:
pytest -s -p no:warnings tests/test_mocks.py

test-utils:
pytest -s -p no:warnings tests/test_utils.py

test-miner:
pytest -s -p no:warnings tests/test_miner.py

test-validator:
pytest -s -p no:warnings tests/test_validator.py

test-all:
pytest -s -p no:warnings tests
pytest -s -p no:warnings tests/test_miner.py tests/test_validator.py
34 changes: 34 additions & 0 deletions config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"mainnet": {
"organic": {
"sample_size": 3,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would that make sense that the sample size of the organic is defined by the requester instead of the miner?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can definitely make a ticket for that - my concern is that there would need to be a hard-coded limit on the organic sample size, otherwise someone could spam the entire network and request 240 miners / call

"timeout": 10
},
"synthetic": {
"sample_size": 10,
"timeout": 40,
"blocks": 3
},
"healthcheck": {
"timeout": 10,
"sample_size": 25,
"blocks": 360
}
},
"testnet": {
"organic": {
"sample_size": 3,
"timeout": 10
},
"synthetic": {
"timeout": 10,
"sample_size": 5,
"blocks": 1
},
"healthcheck": {
"timeout": 6,
"sample_size": 25,
"blocks": 180
}
}
}
77 changes: 0 additions & 77 deletions docker/functions.sh
View file
Open in desktop

This file was deleted.

54 changes: 0 additions & 54 deletions docker/miner/Dockerfile
View file
Open in desktop

This file was deleted.

Loading
Loading