Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search prov dag #26

Open
wants to merge 64 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
edf6cc9
Add js-search dep
Oddant1 Jan 23, 2025
5749eff
Add JSON search
Oddant1 Jan 23, 2025
be96d45
Add search obj
Oddant1 Jan 23, 2025
a1ed100
_uuid
Oddant1 Jan 23, 2025
df08e49
No debug log
Oddant1 Jan 23, 2025
63fd9a2
Use fuse-js
Oddant1 Jan 27, 2025
54800d1
set up json and fuse search
Oddant1 Jan 27, 2025
a88c5b3
Update DAG to read already parsed and stored json
Oddant1 Jan 27, 2025
9366a82
Cleanup
Oddant1 Jan 28, 2025
ff3b6c2
Key entire section on readerModel.provData
Oddant1 Jan 28, 2025
3ab7d55
start adding searchbar
Oddant1 Jan 28, 2025
a9cf814
merge
Oddant1 Jan 29, 2025
1599580
audit fix
Oddant1 Jan 29, 2025
a2b60da
format
Oddant1 Jan 29, 2025
46bbab9
Start refactoring to create nodes while recursing
Oddant1 Jan 29, 2025
149c910
Rewriting prov dag parse
Oddant1 Jan 30, 2025
0a59ce7
format
Oddant1 Jan 30, 2025
4457083
Merge branch 'main' into search-prov-dag
Oddant1 Jan 30, 2025
8c8cefc
Only need to pass along UUID
Oddant1 Jan 30, 2025
91d8168
details -> citations
Oddant1 Jan 30, 2025
ff61500
We have edges and whatnot now yay
Oddant1 Jan 30, 2025
cba5c8b
Get json
Oddant1 Jan 30, 2025
841c053
Cludge collections to prove I understand what was wrong
Oddant1 Jan 30, 2025
f37c391
Massive refactor of recursive code
Oddant1 Jan 31, 2025
9a18de7
Clean up some state
Oddant1 Jan 31, 2025
e46e2c8
Ton of comments
Oddant1 Jan 31, 2025
7ec49ad
Refactor for collections
Oddant1 Jan 31, 2025
de2a168
Create the Fuse search
Oddant1 Feb 1, 2025
ab122a2
Proof of concept for search
Oddant1 Feb 1, 2025
e5e956a
Whitespace
Oddant1 Feb 1, 2025
d1f690b
Combine arrays better
Oddant1 Feb 1, 2025
949ac8c
Buncha cleanup of types
Oddant1 Feb 3, 2025
7919149
Hack to guarantee collection order
Oddant1 Feb 3, 2025
03da943
values not keys
Oddant1 Feb 3, 2025
10245db
Add minimal biMap
Oddant1 Feb 3, 2025
f55e08b
Turn hit json into ids
Oddant1 Feb 3, 2025
fe769d7
Start removing fuse
Oddant1 Feb 4, 2025
72865db
Start reworking json search
Oddant1 Feb 4, 2025
3953baf
Play with no fuse
Oddant1 Feb 4, 2025
13b0a30
wire up basic search
Oddant1 Feb 5, 2025
e37bdb7
format
Oddant1 Feb 5, 2025
687a6f9
Select first result
Oddant1 Feb 5, 2025
5a62b12
Accidentally pasted one word in the middle of another
Oddant1 Feb 5, 2025
10974de
remove debug log
Oddant1 Feb 5, 2025
d8a66f7
Basic support for AND
Oddant1 Feb 5, 2025
1f98427
Format
Oddant1 Feb 5, 2025
02da426
basic key to array
Oddant1 Feb 5, 2025
635c4c3
format
Oddant1 Feb 5, 2025
fdae42c
Helps to have the split character
Oddant1 Feb 5, 2025
db929b0
Cobbled together parser
Oddant1 Feb 18, 2025
f8df587
TODO
Oddant1 Feb 18, 2025
131fc08
Make CSS global
Oddant1 Feb 18, 2025
fd916c5
Start adding flip through search
Oddant1 Feb 18, 2025
c6e40e0
format
Oddant1 Feb 18, 2025
45cfe41
Better flip through search
Oddant1 Feb 18, 2025
3f97b44
Sort hits by row then col
Oddant1 Feb 18, 2025
403eb3e
Default searchIndex to 1
Oddant1 Feb 18, 2025
4b543eb
Only show search buttons if we have hits
Oddant1 Feb 18, 2025
d35aff1
Merge branch 'main' into search-prov-dag
Oddant1 Feb 18, 2025
9896027
searchIndex wraps
Oddant1 Feb 18, 2025
1eb97b8
Specify NodeSingular type
Oddant1 Feb 18, 2025
de101a4
just make it any because importing the types is looking borked
Oddant1 Feb 18, 2025
20935c7
Still need to import cytoscape
Oddant1 Feb 18, 2025
173af39
Re-Focus and Clear buttons
Oddant1 Feb 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -30,6 +30,7 @@
"dependencies": {
"citation-js": "^0.7.16",
"cytoscape": "^3.30.2",
"js-search": "^2.0.1",
"js-yaml": "^3.14.1",
"jszip": "^3.10.1",
"prettier": "^3.3.3",
20 changes: 20 additions & 0 deletions src/app.css
Original file line number Diff line number Diff line change
@@ -33,6 +33,26 @@ p {
font-size: 18px;
}

.roundInput {
@apply border
border-solid
rounded
border-gray-300
mr-auto
mt-auto
pl-2
mb-4;
}

.roundButton {
@apply p-2
rounded-md;
}

.roundButton:hover {
@apply bg-gray-300;
}

a:link {
text-decoration: underline;
cursor: pointer;
175 changes: 143 additions & 32 deletions src/lib/components/Dag.svelte
Original file line number Diff line number Diff line change
@@ -5,8 +5,139 @@
import provenanceModel from "$lib/models/provenanceModel";
import cytoscape from "cytoscape";
import { provSearchStore } from "$lib/scripts/prov-search-store";
let self: HTMLDivElement;
let cy: cytoscape.Core;
// Search syntax something like
//
// param:sampling_depth=1103 AND action=core_metrics AND plugin=boots
//
// If you are searching for a string value that contains = or : or " AND " you
// will need to put that string in quotes
// If the string contains quotes, they will need to be escaped e.g. \"
// If the string contains \" add additional \ e.g. \\"
//
// param/params/parameter/parameters and input as special keys?
// param and co. match if parameters is above terminal key in path
// inputs mathes if inputs is above terminal key in path
// what happens if param is in middle of complex key?
export function searchProvenance(searchValue: string) {
const tokens: Array<Array<any>> = [];
parser(searchValue, tokens);
const hits: Array<Set<string>> = [];
for (const token of tokens) {
hits.push(provenanceModel.searchJSON(token[0], token[1]));
}
let finalHits: Set<string> | Array<string> = hits[0];
for (let i = 1; i < hits.length; i++) {
finalHits = finalHits.intersection(hits[i])
}
finalHits = Array.from(finalHits);
// Sort the hit nodes by row then by col within a given row
finalHits.sort((a, b) => {
let aNode: any = cy.$id(a);
let bNode: any = cy.$id(b);
if (aNode.descendants().length > 0) {
aNode = aNode.descendants()[0];
}
if (bNode.descendants().length > 0) {
bNode = bNode.descendants()[0];
}
console.log(aNode.data())
console.log(bNode.data())
if (aNode.data().row === bNode.data().row) {
return aNode.data().col - bNode.data().col;
}
return aNode.data().row - bNode.data().row;
});
provSearchStore.set({
searchHits: finalHits
});
}
// TODO: Clean this up and make it work better
// sampling_depth=1103 AND plugin=boots
// sampling_depth=1103 AND plugin="boots AND something"
// sampling_depth=1103 AND plugin="boots AND something" AND action=idk
function parser(searchValue: string, tokens: Array<Array<any>>) {
if (searchValue === "") {
return;
}
// The location of the = that separates key=values
const splitIndex = searchValue.indexOf('=');
// Everything before that = is the key
const keys = searchValue.split('=', 1)[0].split(':');
// The indices marking the beginning and end of the value that goes with
// the current key
let startValIndex = splitIndex + 1;
let endValIndex = searchValue.indexOf(' AND ');
let value = '';
// If the value starts with a quote, our value is everything between this
// quote and the next unescaped quote.
if (searchValue[startValIndex] === '"') {
// Increment past the opening quote
startValIndex++;
endValIndex++;
// Search for the next unescaped quote
do {
endValIndex = searchValue.indexOf('"', endValIndex);
} while (searchValue[endValIndex] === '\\')
// If we hit the end of the searchValue they didn't close their quote
if (endValIndex === -1) {
console.log(keys)
console.log("HERE")
// some kinda parse error
}
// Cut the value out
value = searchValue.slice(startValIndex, endValIndex);
// Increment past the next " AND "
endValIndex += 5;
} else {
endValIndex = searchValue.indexOf(" AND ");
if (endValIndex === -1) {
value = searchValue.slice(startValIndex);
endValIndex = searchValue.length;
} else {
value = searchValue.slice(startValIndex, endValIndex);
endValIndex += 5;
}
}
tokens.push([keys, value]);
parser(searchValue.slice(endValIndex), tokens);
}
export function selectSearchHit(hitUUID: string) {
const elem = cy.$id(hitUUID);
elem.select();
cy.center(elem);
// Pan to put the focused node near the top of the viewport
cy.panBy({
x: 0,
y: ((provenanceModel.height - 2) / 2) * -105
})
}
const cytoscapeConfig = {
boxSelectionEnabled: true,
@@ -62,34 +193,15 @@
]
};
async function setActionSelection(uuid: string) {
if (uuid in provenanceModel.collectionMapping) {
// If our uuid is a collectionID we get the uuid of the first element of
// the collection to actually get the provenance action.
uuid = provenanceModel.collectionMapping[uuid][0]['uuid'];
}
function setActionSelection(uuid: string) {
provenanceModel.provTitle = "Action Details";
const selectionData = await provenanceModel.getProvenanceAction(uuid);
const selectionData = provenanceModel.nodeIDToJSON.get(uuid);
_setSelection(selectionData);
}
async function setResultSelection(uuid: string) {
function setResultSelection(uuid: string) {
provenanceModel.provTitle = "Result Details";
const selectionData = await provenanceModel.getProvenanceArtifact(uuid);
_setSelection(selectionData);
}
async function setCollectionSelection(uuid: string) {
const selectionData = {};
provenanceModel.provTitle = "Collection Details";
for (const artifact of provenanceModel.collectionMapping[uuid]) {
selectionData[artifact['key']] = await provenanceModel.getProvenanceArtifact(artifact['uuid']);
}
let selectionData = provenanceModel.nodeIDToJSON.get(uuid);
_setSelection(selectionData);
}
@@ -108,13 +220,18 @@
}
onMount(() => {
// null this out when mounting a new DAG
provSearchStore.set({
searchHits: new Set()
});
// Set this height so we center the DAG based on this height
let displayHeight = (provenanceModel.height + 1) * 105;
self.style.setProperty("height", `${displayHeight}px`);
let lock = false; // used to prevent recursive event storms
let selectedExists = false;
let cy = cytoscape({
cy = cytoscape({
...cytoscapeConfig,
container: document.getElementById("cy"),
elements: provenanceModel.elements
@@ -136,15 +253,9 @@
// nodes as its children. We get the action provenance from whichever
// of its children happens to be first. It doesn't matter which because
// the data for the action itself won't change regardless.
setActionSelection(node.children()[0].data("id"));
setActionSelection(node.data("id"));
} else {
const uuid = node.data("id");
if (uuid in provenanceModel.collectionMapping) {
setCollectionSelection(uuid);
} else {
setResultSelection(uuid);
}
setResultSelection(node.data("id"));
}
const edges = node.edgesTo("node");
27 changes: 4 additions & 23 deletions src/lib/components/Gallery.svelte
Original file line number Diff line number Diff line change
@@ -122,7 +122,7 @@

<h2>Gallery</h2>
<p class="pb-2">Don&apos;t have a QIIME 2 result of your own to view? Try one of these!</p>
<input id="searchInput" placeholder="search" on:input={applySearchFilter}/>
<input class="roundInput" id="searchInput" placeholder="search" on:input={applySearchFilter}/>
{#await getGalleryEntries()}
<h3>Fetching Gallery...</h3>
{:then}
@@ -150,7 +150,7 @@
currentPage--;
}
}}
class="pageButton"
class="roundButton"
>
<svg fill="none"
width="10"
@@ -168,7 +168,7 @@
currentPage++;
}
}}
class="pageButton"
class="roundButton"
>
<svg fill="none"
width="10"
@@ -183,6 +183,7 @@
<div class="ml-auto">
<span>Per Page:&nbsp;</span>
<input
class="roundInput"
id="setCardsPerPage"
type="number"
value={cardsPerPage}
@@ -193,29 +194,9 @@
</div>

<style lang="postcss">
input {
@apply border
border-solid
rounded
border-gray-300
mr-auto
mt-auto
pl-2
mb-4;
}
#pageControls {
@apply grid
grid-cols-3
pt-4;
}
.pageButton {
@apply p-2
rounded-md;
}
button:hover {
@apply bg-gray-300;
}
</style>
118 changes: 104 additions & 14 deletions src/lib/components/Provenance.svelte
Original file line number Diff line number Diff line change
@@ -1,24 +1,114 @@
<script lang="ts">
import "../../app.css";
import Panel from "$lib/components/Panel.svelte";
import JSONTree from "svelte-json-tree";
import Dag from "./Dag.svelte";
import provenanceModel from "$lib/models/provenanceModel";
import { provSearchStore } from "$lib/scripts/prov-search-store";
let DAG: Dag;
let value: string = '';
let searchIndex = 1;
let searchHits: Array<string>;
provSearchStore.subscribe((value) => {
searchHits = value.searchHits
});
$: {
if (DAG !== undefined && searchHits[searchIndex - 1] !== undefined) {
DAG.selectSearchHit(searchHits[searchIndex - 1]);
}
}
</script>

{#key $provenanceModel.uuid}
<Dag />
<Dag bind:this={DAG}/>
{/key}
{#key $provenanceModel.provData}
<div>
<form on:submit|preventDefault>
<label>
Search Provenance:
<input class="roundInput" bind:value />
</label>
<button on:click={() => {
searchIndex = 1;
DAG.searchProvenance(value)
}}>GO</button>
</form>
{#if searchHits.length > 0}
<div class="mx-auto">
<button
on:click={() => {
if (searchIndex > 1) {
searchIndex--;
} else {
searchIndex = searchHits.length;
}
}}
class="roundButton"
>
<svg fill="none"
width="10"
height="10">
<path
stroke-width="3"
stroke="rgb(119, 119, 119)"
d="m8 0L3 5a0,2 0 0 1 1,1M3 5L8 10"/>
</svg>
</button>
{searchIndex}/{searchHits.length}
<button
on:click={() => {
if (searchIndex < searchHits.length) {
searchIndex++;
} else {
searchIndex = 1;
}
}}
class="roundButton"
>
<svg fill="none"
width="10"
height="10">
<path
stroke-width="3"
stroke="rgb(119, 119, 119)"
d="m3 0L8 5a0,2 0 0 1 1,1M8 5L3 10"/>
</svg>
</button>
<button
on:click={() => DAG.selectSearchHit(searchHits[searchIndex - 1])}
class="roundButton"
>
Re-Focus
</button>
<button
on:click={() => {
searchHits = [];
value = "";
}}
class="roundButton"
>
Clear
</button>
</div>
{/if}
<Panel header={$provenanceModel.provTitle}>
{#if provenanceModel.provData !== undefined}
<div class="JSONTree">
<JSONTree
value={provenanceModel.provData}
defaultExpandedLevel={100}
shouldShowPreview={false}
/>
</div>
{:else}
<p>Click on an element of the Provenance Graph to learn more</p>
{/if}
</Panel>
</div>
{/key}
<Panel header={$provenanceModel.provTitle}>
{#if $provenanceModel.provData !== undefined}
<div class="JSONTree">
<JSONTree
value={provenanceModel.provData}
defaultExpandedLevel={100}
shouldShowPreview={false}
/>
</div>
{:else}
<p>Click on an element of the Provenance Graph to learn more</p>
{/if}
</Panel>
672 changes: 437 additions & 235 deletions src/lib/models/provenanceModel.ts

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/lib/models/readerModel.ts
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@
import JSZip from "jszip";

import { handleError } from "$lib/scripts/util";

import loading from "$lib/scripts/loading";
import citationsModel from "$lib/models/citationsModel";
import provenanceModel from "$lib/models/provenanceModel";
32 changes: 32 additions & 0 deletions src/lib/scripts/biMap.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Two way map class minimal implementation for what is currently needed
export default class BiMap {
keyToValue;
valueToKey;

constructor() {
this.keyToValue = new Map();
this.valueToKey = new Map();
}

set(key: any, value: any) {
this.keyToValue.set(key, value);
this.valueToKey.set(value, key);
}

get(key: any): any {
return this.keyToValue.get(key);
}

getKey(value: any): any {
return this.valueToKey.get(value);
}

values(): any {
return this.keyToValue.values();
}

clear() {
this.keyToValue.clear();
this.valueToKey.clear();
}
}
7 changes: 7 additions & 0 deletions src/lib/scripts/prov-search-store.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import { writable } from "svelte/store";

export const provSearchStore = writable<{
searchHits: Array<string>;
}>({
searchHits: [],
});
25 changes: 25 additions & 0 deletions src/lib/scripts/util.ts
Original file line number Diff line number Diff line change
@@ -114,3 +114,28 @@ export function getScrollBarWidth() {
// Return difference in widths, this is the width of the scrollbar
return withoutScrollWidth - withScrollWidth;
}

export function getAllObjectKeysRecursively(
targetObject: object,
currentkeyAccumulator: Array<string>,
globalKeySet: Array<Array<string>>,
) {
if (targetObject !== null && targetObject !== undefined) {
for (const key of Object.keys(targetObject)) {
const newKeyAccumulator = [...currentkeyAccumulator, key];

// Some terminal values, such as the start and end times, will parse as objects, but they do not have keys of their own
const next = targetObject[key as keyof object];
if (
typeof next === "object" &&
next !== null &&
next !== undefined &&
Object.keys(next).length !== 0
) {
getAllObjectKeysRecursively(next, newKeyAccumulator, globalKeySet);
} else {
globalKeySet.push(newKeyAccumulator);
}
}
}
}