Sample data not loaded correctly?

[hugokoopmans]

Hi guys,
I tried to create a sample application today, straight from the readme.
Al fine untill the data load step...
My setup is that I have a ML8 server and a laptop. I installed a new project with slush on my laptop.
Added a hugo.properties to reflect my remote ML8 server.
Bootstrapped and deployed the app, with browser connected to the app endpoint, app shows up!

Now loading the sample data I got into trouble.

I did this:

./ml hugo mlcp import  -username admin -password admin -host ml8-base-ml1 -port 8051 -input_file_path data -document_type text -transform_module /transform/from-json.xqy  -transform_namespace "http://marklogic.com/transform/to-json"  -transform_function transform

15/06/29 21:18:38 INFO contentpump.ContentPump: Hadoop library version: 2.0.0-mr1-cdh4.3.0
15/06/29 21:18:38 INFO contentpump.LocalJobRunner: Content type: TEXT
15/06/29 21:18:38 WARN util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
15/06/29 21:18:38 INFO input.FileInputFormat: Total input paths to process : 25
15/06/29 21:18:39 WARN mapreduce.ContentWriter: XDMP-MODNOTFOUND: (err:XQST0059) Module /transform/from-json.xqy not found
15/06/29 21:18:39 WARN mapreduce.ContentWriter: XDMP-MODNOTFOUND: (err:XQST0059) Module /transform/from-json.xqy not found

Now in the documentation it says there should be a from-json.xqy in the /transform dir ... I looked for 'from-json.xqy' in /transforms but only found a 'to-json.xqy' there... seems these transforms are derived from the template that uses xslt to generate these transforms?

The to-json looks like this:

xquery version "1.0-ml";
module namespace trns = "http://marklogic.com/transform/to-json";

import module namespace json="http://marklogic.com/xdmp/json" at "/MarkLogic/json/json.xqy";

declare function trns:transform(
  $content as map:map,
  $context as map:map
) as map:map*
{
  map:put(
    $content,
    'value',
    document { json:transform-from-json(map:get($content, 'value')) }
  ),
  $content
};

intuition/ignorance mode from here

I changed the transform-from-json to transform-to-json in this to-json.xqy and the mlcp line to use transform/to-json.xqy ... now the data loads but no search results

should there be a from-json.xqy in /transforms ?

[hugokoopmans]

now I have two observations:

1. Starting gulp (before gulp server) it throws an error not sure if this is relevant.

hugo@vaio ~/git/slush-dikw-node/slush-dikw-node $ gulp
[21:52:06] Using gulpfile ~/git/slush-dikw-node/slush-dikw-node/gulpfile.js
[21:52:06] Starting 'jshint'...
[21:52:06] Finished 'jshint' after 145 ms
[21:52:06] Starting 'less'...
[21:52:06] Starting 'scripts'...
[21:52:06] Starting 'watch'...
[21:52:09] Finished 'watch' after 3.14 s
/home/hugo/git/slush-dikw-node/slush-dikw-node/ui/app/common/common.js: line 5, col 3, Missing "use strict" statement.

1 error
/home/hugo/git/slush-dikw-node/slush-dikw-node/ui/app/create/create-ctrl.js: line 45, col 0, Identifier 'toolbar_full' is not in camel case.

1 error
[21:52:11] Finished 'less' after 4.84 s
[21:52:11] Finished 'scripts' after 5.14 s
[21:52:11] Starting 'default'...
[21:52:11] Finished 'default' after 7.9 μs

2. If I inspect the app it gives me 3 errors all the same, complaining it cannot find something

gd-ui-jsonexplorer.css
/bower_components/ng-json-explorer/src

In the header:

Remote Address:127.0.0.1:9020
Request URL:http://localhost:9020/bower_components/ng-json-explorer/src/gd-ui-jsonexplorer.css
Request Method:GET
Status Code:404 Not Found

And indeed there is no gd-ui-jsonexplorer.css there... there is a angular-json-explorer.js however...

[grtjn]

Sounds like you are missing latest bug fixes. You need to install slush-marklogic-node directly from github as described here: #68

Install, and then recreate the project..

[grtjn]

That will also fix the json-explorer issue. You can ignore the jshint warning about js syntax..

[hugokoopmans]

hi grtjn,
tx for your quick response. bare with me here... so I need to use/clone from the dev branch correct?

[hugokoopmans]

sorry I seem not to be able to find a dev branch on this project? Latest is release 0.1.1 ?

[grtjn]

There is no dev branch, just master. Last tag and npm release is 0.1.1, but master contains latter bugfixes. Use this command to install directly from github:

sudo npm install -g git://github.com/marklogic/slush-marklogic-node.git

After that, recreate the project with slush marklogic-node..

[hugokoopmans]

UPDATE I used roxy dev branche in the slection when setting up the app

hmmm application setup again but still no from-json.xqy in the transforms dir only xsl files so:

./import-sample-data.sh -e hugo
Cleaning content first
Cleaning slush-dikw-node-content on ml8-base-ml1

Uploading sample data

15/06/30 10:47:17 INFO contentpump.ContentPump: Hadoop library version: 2.0.0-mr1-cdh4.3.0
15/06/30 10:47:17 INFO contentpump.LocalJobRunner: Content type: TEXT
15/06/30 10:47:17 WARN util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
15/06/30 10:47:17 INFO input.FileInputFormat: Total input paths to process : 25
15/06/30 10:47:22 WARN mapreduce.ContentWriter: XDMP-MODNOTFOUND: (err:XQST0059) Module /transform/from-json.xqy not found

Seems the transform xqy files are not generated from template?

[hugokoopmans]

I looked at from-json.xsl and to-json.xsl they are quite different. If I understand correctly roxy creates the transforms from these xsl files right? Seems roxy only created a transform from the to-json.xsl actually and not from the from-json.xsl ?

[grtjn]

Did you run the sudo command I gave above before rerunning slush marklogic-node to create a new project?

[grtjn]

And if running against MarkLogic 8, you won't be needing that json transform at ingest. Just skip the transform..

[hugokoopmans]

yep did sudo npm install from github, made changes to port etc app came up all right with gulp etc...

ok skipping the transform did it... now the data loads as txt ...

playing around with the mlcp commands will make things tidy I guess...

./ml hugo mlcp import  -username admin -password admin -host ml8-base-ml1 -port 8051 -input_file_path sample-data -document_type text

[grtjn]

You may not have noticed, but the latest README doesn't mention the transform options anymore:

https://github.com/marklogic/slush-marklogic-node/blob/master/app/templates/README.mdown

Also note that when running mlcp through Roxy, username, password, host and port are superfluous..