Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next Range Calculation error #4840

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

parrjd
Copy link
Contributor

@parrjd parrjd commented Sep 4, 2024

NextRange is stored in ldap database as a Hex string

mIncrementNo is provided by CertificateRepository.java and is converted dec to BigInt with mRadix

nextRangeNo Needs to be treated the same as mIncrementNo so that add and subtract work correctly and produce the correct values and match up with what will be put into CS.cfg to prevent large gaps with sequential Serial Managment.

… to Dec BigInteger to add increment correctly
Copy link

sonarcloud bot commented Sep 4, 2024

@fmarco76
Copy link
Member

fmarco76 commented Sep 11, 2024

@parrjd I have tried to reproduce the next range calculation problem this PR tries to fix.
I have a new CA instance based on master branch code and I have configured the range size to be 8, from CS.cfg:

dbs.beginRequestNumber=1
dbs.beginSerialNumber=1
dbs.endRequestNumber=8
dbs.endSerialNumber=8
dbs.enableSerialManagement=true
dbs.requestIncrement=8
dbs.serialIncrement=8

I have created 9 certificate to get to the new range and in the DS I have:

[root@pki pki]# ldapsearch -H ldap://ds.example.com:3389 -D "cn=Directory Manager" -w Secret.123 -s base -b ou=certificateRepository,ou=ca,dc=ca,dc=pki,dc=example,dc=com
# extended LDIF
#
# LDAPv3
# base <ou=certificateRepository,ou=ca,dc=ca,dc=pki,dc=example,dc=com> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

# certificateRepository, ca, ca.pki.example.com
dn: ou=certificateRepository,ou=ca,dc=ca,dc=pki,dc=example,dc=com
serialno: 011
ou: certificateRepository
objectClass: top
objectClass: repository
nextRange: 25

# search result
search: 2
result: 0 Success

and

[root@pki pki]# ldapsearch -H ldap://ds.example.com:3389 -D "cn=Directory Manager" -w Secret.123 -b ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
# extended LDIF
#
# LDAPv3
# base <ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# certificateRepository, ranges, ca.pki.example.com
dn: ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
ou: certificateRepository
objectClass: top
objectClass: organizationalUnit

# 9, certificateRepository, ranges, ca.pki.example.com
dn: cn=9,ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 9
endRange: 16
cn: 9
host: pki.example.com
SecurePort: 8443

# 17, certificateRepository, ranges, ca.pki.example.com
dn: cn=17,ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 17
endRange: 24
cn: 17
host: pki.example.com
SecurePort: 8443

# search result
search: 2
result: 0 Success

Therefore, the nextRange attribute in DS is in decimal and the range is correctly calculated. I have reported also the ranges and all the values are in decimal and correct.

The log information just after the change is also reported correctly:

2024-09-11 13:07:35 [main] INFO: Repository: Updating nextRange from 17 to 25

The only place where I found a problem is in the CS.cfg because after the update I get:

dbs.beginReplicaNumber=1
dbs.beginRequestNumber=17
dbs.beginSerialNumber=17
dbs.enableSerialManagement=true
dbs.endReplicaNumber=100
dbs.endRequestNumber=24
dbs.endSerialNumber=1e

The serial number end range is in hex format but the request end range is in decimal so the problem is specific to the serial number.
Actually, the CertificateRepository set itself to hex while the RequestRepository to decimal but I have not investigated the reason of this difference yet.

If you have different result could describe how to configure the CA to get the same behaviour?

@parrjd
Copy link
Contributor Author

parrjd commented Sep 11, 2024

@parrjd I have tried to reproduce the next range calculation problem this PR tries to fix. I have a new CA instance based on master branch code and I have configured the range size to be 8, from CS.cfg:

dbs.beginRequestNumber=1
dbs.beginSerialNumber=1
dbs.endRequestNumber=8
dbs.endSerialNumber=8
dbs.enableSerialManagement=true
dbs.requestIncrement=8
dbs.serialIncrement=8

I have created 9 certificate to get to the new range and in the DS I have:

[root@pki pki]# ldapsearch -H ldap://ds.example.com:3389 -D "cn=Directory Manager" -w Secret.123 -s base -b ou=certificateRepository,ou=ca,dc=ca,dc=pki,dc=example,dc=com
# extended LDIF
#
# LDAPv3
# base <ou=certificateRepository,ou=ca,dc=ca,dc=pki,dc=example,dc=com> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

# certificateRepository, ca, ca.pki.example.com
dn: ou=certificateRepository,ou=ca,dc=ca,dc=pki,dc=example,dc=com
serialno: 011
ou: certificateRepository
objectClass: top
objectClass: repository
nextRange: 25

# search result
search: 2
result: 0 Success

and

[root@pki pki]# ldapsearch -H ldap://ds.example.com:3389 -D "cn=Directory Manager" -w Secret.123 -b ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
# extended LDIF
#
# LDAPv3
# base <ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# certificateRepository, ranges, ca.pki.example.com
dn: ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
ou: certificateRepository
objectClass: top
objectClass: organizationalUnit

# 9, certificateRepository, ranges, ca.pki.example.com
dn: cn=9,ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 9
endRange: 16
cn: 9
host: pki.example.com
SecurePort: 8443

# 17, certificateRepository, ranges, ca.pki.example.com
dn: cn=17,ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 17
endRange: 24
cn: 17
host: pki.example.com
SecurePort: 8443

# search result
search: 2
result: 0 Success

Therefore, the nextRange attribute in DS is in decimal and the range is correctly calculated. I have reported also the ranges and all the values are in decimal and correct.

The log information just after the change is also reported correctly:

2024-09-11 13:07:35 [main] INFO: Repository: Updating nextRange from 17 to 25

The only place where I found a problem is in the CS.cfg because after the update I get:

dbs.beginReplicaNumber=1
dbs.beginRequestNumber=17
dbs.beginSerialNumber=17
dbs.enableSerialManagement=true
dbs.endReplicaNumber=100
dbs.endRequestNumber=24
dbs.endSerialNumber=1e

The serial number end range is in hex format but the request end range is in decimal so the problem is specific to the serial number. Actually, the CertificateRepository set itself to hex while the RequestRepository to decimal but I have not investigated the reason of this difference yet.

If you have different result could describe how to configure the CA to get the same behaviour?

you need to use a larger increment then 10 anything above 10 will produce the error

11 Hex is 17 Dec. With an increment of 11 your first next range would be 9 - 26 since it is adding a converted 11 hex to 9, and what is stored in the CS,cfg would be 1A causing a Gab from 1A - 27 serials in hex and the next being

@fmarco76
Copy link
Member

you need to use a larger increment then 10 anything above 10 will produce the error

11 Hex is 17 Dec. With an increment of 11 your first next range would be 9 - 26 since it is adding a converted 11 hex to 9, and what is stored in the CS,cfg would be 1A causing a Gab from 1A - 27 serials in hex and the next being

I have tested with range of size 12 (for both certificates and requests) and next range is in decimal in the DS entry however the range is wrong because the increment is read as hex from the CS.cfg so I have increments of 12 for the requests and 18 (0x12 ) for the certificate:

# 13, requests, ranges, ca.pki.example.com
dn: cn=13,ou=requests,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 13
endRange: 24
cn: 13
host: pki.example.com
SecurePort: 8443

# 13, certificateRepository, ranges, ca.pki.example.com
dn: cn=13,ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 13
endRange: 30
cn: 13
host: pki.example.com

There is a problem of conversion but I do not think this PR is fixing the right place. I'll do additional investigation to verify.

@parrjd
Copy link
Contributor Author

parrjd commented Sep 13, 2024

@fmarco76
The Values in the CS.cfg for serial for the range and the next range are stored in Hex and Not Dec using mRadix.

public void setMinSerialConfig() throws EBaseException {
EngineConfig cs = engine.getConfig();
DatabaseConfig dbConfig = dbSubsystem.getDBConfigStore();
String serial = mMinSerialNo.toString(mRadix);
logger.debug("Repository: Setting min serial number: " + serial);
dbConfig.putString(minSerialName, serial);
cs.commit(false);
}
/**
* Sets maximum serial number limit in config file
*
* @exception EBaseException failed to set
*/
public void setMaxSerialConfig() throws EBaseException {
EngineConfig cs = engine.getConfig();
DatabaseConfig dbConfig = dbSubsystem.getDBConfigStore();
String serial = mMaxSerialNo.toString(mRadix);
logger.debug("Repository: Setting max serial number: " + serial);
dbConfig.putString(maxSerialName, serial);
cs.commit(false);
}
/**
* Sets minimum serial number limit for next range in config file
*
* @exception EBaseException failed to set
*/
public void setNextMinSerialConfig() throws EBaseException {
EngineConfig cs = engine.getConfig();
DatabaseConfig dbConfig = dbSubsystem.getDBConfigStore();
if (mNextMinSerialNo == null) {
logger.debug("Repository: Removing next min number");
dbConfig.remove(nextMinSerialName);
} else {
String serial = mNextMinSerialNo.toString(mRadix);
logger.debug("Repository: Setting next min number: " + serial);
dbConfig.putString(nextMinSerialName, serial);
}
cs.commit(false);
}
/**
* Sets maximum serial number limit for next range in config file
*
* @exception EBaseException failed to set
*/
public void setNextMaxSerialConfig() throws EBaseException {
EngineConfig cs = engine.getConfig();
DatabaseConfig dbConfig = dbSubsystem.getDBConfigStore();
if (mNextMaxSerialNo == null) {
logger.debug("Repository: Removing next max number");
dbConfig.remove(nextMaxSerialName);
} else {
String serial = mNextMaxSerialNo.toString(mRadix);
logger.debug("Repository: Setting next max number: " + serial);
dbConfig.putString(nextMaxSerialName, serial);
}
cs.commit(false);
}

The Value stored in the Database in the PROP_NEXT_RANGE attribute should be in HEX as it matches the beginning of the Next Range values stored in the CS.cfg

I have seen both the values in the DB and the values in CS.cfg being Hex with letters a-f in the value

Calculation for mNextMax Serial number uses mRadix

logger.debug("Repository: Next min serial number: " + mNextMinSerialNo.toString(mRadix));
mNextMaxSerialNo = mNextMinSerialNo.add(mIncrementNo).subtract(BigInteger.ONE);
numsAvail = numsAvail.add(mIncrementNo);

Where as the Calculation for what is stored in the database does not.

String nextRange = attr.getStringValues().nextElement();
BigInteger nextRangeNo = new BigInteger(nextRange);
BigInteger newNextRangeNo = nextRangeNo.add(mIncrementNo);
String newNextRange = newNextRangeNo.toString();
String endRange = newNextRangeNo.subtract(BigInteger.ONE).toString();

Based on this section of code it looks like ReplicaID and requests are treated as base 10 (dec), and SerialNo are treated as base 16 (hex)

if (type.equals("replicaId")) {
logger.debug("UpdateNumberRange: Checking replica number ranges");
repo.checkRanges();
}
int radix = 10;
String endNumConfig = null;
String cloneNumConfig = null;
String nextEndConfig = null;
if (type.equals("request")) {
radix = 10;
endNumConfig = "endRequestNumber";
cloneNumConfig = "requestCloneTransferNumber";
nextEndConfig = "nextEndRequestNumber";
} else if (type.equals("serialNo")) {
radix = 16;
endNumConfig = "endSerialNumber";
cloneNumConfig = "serialCloneTransferNumber";
nextEndConfig = "nextEndSerialNumber";
} else if (type.equals("replicaId")) {
radix = 10;
endNumConfig = "endReplicaNumber";
cloneNumConfig = "replicaCloneTransferNumber";
nextEndConfig = "nextEndReplicaNumber";
}

@fmarco76
Copy link
Member

@parrjd Yes, you are correct. Actually, internal operations are on BigInteger so it does not matter but serial and keys (in KRA subsystem) use mRadix to reead/write from CS.cfg.

https://github.com/dogtagpki/pki/blob/master/base/ca/src/main/java/com/netscape/cmscore/dbs/CertificateRepository.java#L135-L181

Differently, the SubsystemRangeUpdateCLI read from CS.cfg in decimal for all the ranges and write them in DS, this is the reason the initial nextRange match the value written in CS.cfg.

This PR could solve some of the error but there are several limitation I am investigate. In the documentation I did not find what format the ranges should have. I would investigate with other if it is possible to use the same format for both. This is trivial for new instance but for running instances with ranges created this is problematic, I have problem updating the value to be all hex or all decimal but I am working on that.

@parrjd
Copy link
Contributor Author

parrjd commented Sep 13, 2024

What we have done as an interim break fix to keep it from leaving large gaps is we are watching for a role and when it roles, we are pulling the values from the DS Range entry and comparing it to the CS.cfg and if the end does not match the DS Range entry we are updating the CS.cfg to match and then restarting the CA instance.

13, certificateRepository, ranges, ca.pki.example.com

dn: cn=13,ou=certificateRepository,ou=ranges,dc=ca,dc=pki,dc=example,dc=com
objectClass: top
objectClass: pkiRange
beginRange: 13
endRange: 30
cn: 13

dbs.beginSerialNumber=13
dbs.endSerialNumber=24

So we are updating dbs.endSerialNumber= to match the endRange: value from the DS instance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants