Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix failed to create model archive #1508

Merged
merged 5 commits into from
Mar 19, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,15 @@ public static ModelArchive downloadModel(
}

if (new File(url).isDirectory()) {
// handle the case that the input url is a directory.
// the input of url is "/xxx/model_store/modelXXX" or
// "xxxx/yyyyy/modelXXX".
return load(url, new File(url), false);
} else if (modelLocation.exists()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lxning how would we handle the case if there are multiple mar files in a directory? are we relying on user to have only one mar file in the /xxx/model_store/modelXXX directory? In this case we need to make sure we have documented it clearly.

Do you we have any documentation on it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understood this PR as no longer needing a model_store and just letting users link to various model files directy

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In model_store dir, existing code ignores the mar files in the subdir (ie. /xxx/model_store/modelXXX).

// handle the case that "/xxx/model_store/modelXXX" is directory.
// the input of url is modelXXX when torchserve is started
// with snapshot or with parameter --models modelXXX
return load(url, modelLocation, false);
}

throw new ModelNotFoundException("Model not found at: " + url);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ public void archiveTest() throws ModelException, IOException, DownloadArchiveExc
ModelNotFoundException.class,
() ->
ModelArchive.downloadModel(
ALLOWED_URLS_LIST, "src/test/resources/", "models"));
ALLOWED_URLS_LIST, "src/test/resources", "noop_no_archive"));

archive.clean();
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"runtime": "python",
"model": {
"modelName": "noop",
"serializedFile": "model.pt",
"handler": "service.py"
},
"modelServerVersion": "1.0",
"implementationVersion": "1.0",
"specificationVersion": "1.0"
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to store a model.pt in version control? Is it just an empty file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, noop is an example. model.pt is an empty file. Manifest requires it to pass validation.

Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@


"""
NoopService defines a no operational model handler.
"""
import logging
import time


class NoopService(object):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow why a NoopService is needed, it seems very similar to the base handler? Could you please explain at a high level in the github issue the design of this PR

Copy link
Collaborator Author

@lxning lxning Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example noop_no_archive_no_version is copied from original noop_no_archive. I did slightly change (ie remove modelversion in manifest.json) for unit test the default manifest bug fixing.

"""
Noop Model handler implementation.

Extend from BaseModelHandler is optional
"""

def __init__(self):
self._context = None
self.initialized = False

def initialize(self, context):
"""
Initialize model. This will be called during model loading time

:param context: model server context
:return:
"""
self.initialized = True
self._context = context

@staticmethod
def preprocess(data):
"""
Transform raw input into model input data.

:param data: list of objects, raw input from request
:return: list of model input data
"""
return data

@staticmethod
def inference(model_input):
"""
Internal inference methods

:param model_input: transformed model input data
:return: inference results
"""
return model_input

@staticmethod
def postprocess(model_output):
return ["OK"] * len(model_output)

def handle(self, data, context):
"""
Custom service entry point function.

:param context: model server context
:param data: list of objects, raw input from request
:return: list of outputs to be send back to client
"""
# Add your initialization code here
properties = context.system_properties
server_name = properties.get("server_name")
server_version = properties.get("server_version")
model_dir = properties.get("model_dir")
gpu_id = properties.get("gpu_id")
batch_size = properties.get("batch_size")

logging.debug("server_name: {}".format(server_name))
logging.debug("server_version: {}".format(server_version))
logging.debug("model_dir: {}".format(model_dir))
logging.debug("gpu_id: {}".format(gpu_id))
logging.debug("batch_size: {}".format(batch_size))
try:
preprocess_start = time.time()
data = self.preprocess(data)
inference_start = time.time()
data = self.inference(data)
postprocess_start = time.time()
data = self.postprocess(data)
end_time = time.time()

context.set_response_content_type(0, "text/plain")

content_type = context.request_processor[0].get_request_property("Content-Type")
logging.debug("content_type: {}".format(content_type))

metrics = context.metrics
metrics.add_time("PreprocessTime", round((inference_start - preprocess_start) * 1000, 2))
metrics.add_time("InferenceTime", round((postprocess_start - inference_start) * 1000, 2))
metrics.add_time("PostprocessTime", round((end_time - postprocess_start) * 1000, 2))
return data
except Exception as e:
logging.error(e, exc_info=True)
context.request_processor[0].report_status(500, "Unknown inference error.")
return ["Error {}".format(str(e))] * len(data)


_service = NoopService()


def handle(data, context):
if not _service.initialized:
_service.initialize(context)

if data is None:
return None

return _service.handle(data, context)
Original file line number Diff line number Diff line change
Expand Up @@ -169,25 +169,30 @@ private ModelArchive createModelArchive(
configManager.getModelStore(),
url,
s3SseKms);
Manifest.Model model = archive.getManifest().getModel();
if (modelName == null || modelName.isEmpty()) {
if (archive.getModelName() == null || archive.getModelName().isEmpty()) {
archive.getManifest().getModel().setModelName(defaultModelName);
model.setModelName(defaultModelName);
}
} else {
archive.getManifest().getModel().setModelName(modelName);
model.setModelName(modelName);
}

if (runtime != null) {
archive.getManifest().setRuntime(runtime);
}

if (handler != null) {
archive.getManifest().getModel().setHandler(handler);
model.setHandler(handler);
} else if (archive.getHandler() == null || archive.getHandler().isEmpty()) {
archive.getManifest().getModel().setHandler(configManager.getTsDefaultServiceHandler());
model.setHandler(configManager.getTsDefaultServiceHandler());
}

archive.getManifest().getModel().setEnvelope(configManager.getTsServiceEnvelope());
model.setEnvelope(configManager.getTsServiceEnvelope());

if (model.getModelVersion() == null) {
model.setModelVersion("1.0");
}

archive.validate();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -419,7 +419,7 @@ public void testNoopPrediction() throws InterruptedException {
@Test(
alwaysRun = true,
dependsOnMethods = {"testNoopPrediction"})
public void testLoadModelWithNakedDirModelArchive() throws InterruptedException {
public void testLoadModelWithNakedDirNoVersionModelArchive() throws InterruptedException {
String operatingSystem = System.getProperty("os.name").toLowerCase();
if (!operatingSystem.contains("win")) {
Channel channel = TestUtils.getManagementChannel(configManager);
Expand All @@ -429,6 +429,38 @@ public void testLoadModelWithNakedDirModelArchive() throws InterruptedException
DefaultFullHttpRequest req =
new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/models");
req.headers().add("Content-Type", "application/json");
req.content()
.writeCharSequence(
"{'url':'"
+ configManager.getModelStore()
+ "/noop_no_archive_no_version/noop/"
+ "', 'model_name':'noop', 'initial_workers':'1', 'synchronous':'true'}",
CharsetUtil.UTF_8);
HttpUtil.setContentLength(req, req.content().readableBytes());
channel.writeAndFlush(req);
TestUtils.getLatch().await();

StatusResponse resp =
JsonUtils.GSON.fromJson(TestUtils.getResult(), StatusResponse.class);
Assert.assertEquals(
resp.getStatus(),
"Model \"noop\" Version: 1.0 registered with 1 initial workers");
}
}

@Test(
alwaysRun = true,
dependsOnMethods = {"testLoadModelWithNakedDirNoVersionModelArchive"})
public void testLoadModelWithNakedDirModelArchive() throws InterruptedException {
String operatingSystem = System.getProperty("os.name").toLowerCase();
if (!operatingSystem.contains("win")) {
Channel channel = TestUtils.getManagementChannel(configManager);
testUnregisterModel("noop", "1.0");
TestUtils.setResult(null);
TestUtils.setLatch(new CountDownLatch(1));
DefaultFullHttpRequest req =
new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/models");
req.headers().add("Content-Type", "application/json");
req.content()
.writeCharSequence(
"{'url':'"
Expand Down