Skip to content

Commit

Permalink
deploy: 5cd8dc7
Browse files Browse the repository at this point in the history
  • Loading branch information
pomadchin committed Jul 2, 2024
1 parent 3e8a30b commit 8aa4151
Show file tree
Hide file tree
Showing 7 changed files with 21 additions and 21 deletions.
4 changes: 2 additions & 2 deletions Cats.html
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ <h2 id="effect-suspension-in-typed-datasets" class="section"><a class="anchor-li
</span><span class="identifier">count</span><span> &lt;- </span><span class="identifier">typedDs</span><span>.</span><span class="identifier">count</span><span>[</span><span class="type-name">Action</span><span>]()
} </span><span class="keyword">yield</span><span> (</span><span class="identifier">sample</span><span>, </span><span class="identifier">count</span><span>)
</span><span class="comment">// result: Action[(Seq[(Int, String)], Long)] = Kleisli(
// cats.data.Kleisli$$Lambda$11673/0x0000000803398040@529321b4
// cats.data.Kleisli$$Lambda$11614/0x000000080330d840@157d8902
// )</span></code></pre>
<p>As with <code>Job</code>, note that nothing has been run yet. The effect has been properly suspended. To
run our program, we must first supply the <code>SparkSession</code> to the <code>ReaderT</code> layer and then
Expand All @@ -193,7 +193,7 @@ <h3 id="convenience-methods-for-modifying-spark-thread-local-variables" class="s
}
} </span><span class="keyword">yield</span><span> </span><span class="identifier">r</span><span>
</span><span class="comment">// resultWithDescription: Action[(Seq[(Int, String)], Long)] = Kleisli(
// cats.data.Kleisli$$$Lambda$13094/0x0000000803885040@63b314e7
// cats.data.Kleisli$$$Lambda$13124/0x00000008038d2840@2fa15da1
// )
</span><span>
</span><span class="identifier">resultWithDescription</span><span>.</span><span class="identifier">run</span><span>(</span><span class="identifier">spark</span><span>).</span><span class="identifier">unsafeRunSync</span><span>()
Expand Down
2 changes: 1 addition & 1 deletion FeatureOverview.html
Original file line number Diff line number Diff line change
Expand Up @@ -716,7 +716,7 @@ <h2 id="user-defined-functions" class="section"><a class="anchor-link left" href
</span><span class="comment">// priceModifier: (String, Double) =&gt; Double = &lt;function2&gt;
</span><span>
</span><span class="keyword">val</span><span> </span><span class="identifier">udf</span><span> = </span><span class="identifier">aptTypedDs</span><span>.</span><span class="identifier">makeUDF</span><span>(</span><span class="identifier">priceModifier</span><span>)
</span><span class="comment">// udf: (frameless.TypedColumn[Apartment, String], frameless.TypedColumn[Apartment, Double]) =&gt; frameless.TypedColumn[Apartment, Double] = frameless.functions.Udf$$Lambda$14191/0x0000000803ce1840@7c973009
</span><span class="comment">// udf: (frameless.TypedColumn[Apartment, String], frameless.TypedColumn[Apartment, Double]) =&gt; frameless.TypedColumn[Apartment, Double] = frameless.functions.Udf$$Lambda$14170/0x0000000803caf040@200c7081
</span><span>
</span><span class="keyword">val</span><span> </span><span class="identifier">aptds</span><span> = </span><span class="identifier">aptTypedDs</span><span> </span><span class="comment">// For shorter expressions
// aptds: TypedDataset[Apartment] = [city: string, surface: int ... 2 more fields]
Expand Down
8 changes: 4 additions & 4 deletions Injection.html
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ <h2 id="example" class="section"><a class="anchor-link left" href="#example"><i
</span><span class="comment">// people: Seq[Person] = List(
// Person(
// 42,
// java.util.GregorianCalendar[time=1718499806448,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id=&quot;Etc/UTC&quot;,offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2024,MONTH=5,WEEK_OF_YEAR=25,WEEK_OF_MONTH=4,DAY_OF_MONTH=16,DAY_OF_YEAR=168,DAY_OF_WEEK=1,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=1,HOUR_OF_DAY=1,MINUTE=3,SECOND=26,MILLISECOND=448,ZONE_OFFSET=0,DST_OFFSET=0]
// java.util.GregorianCalendar[time=1719924507663,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id=&quot;Etc/UTC&quot;,offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2024,MONTH=6,WEEK_OF_YEAR=27,WEEK_OF_MONTH=1,DAY_OF_MONTH=2,DAY_OF_YEAR=184,DAY_OF_WEEK=3,DAY_OF_WEEK_IN_MONTH=1,AM_PM=1,HOUR=0,HOUR_OF_DAY=12,MINUTE=48,SECOND=27,MILLISECOND=663,ZONE_OFFSET=0,DST_OFFSET=0]
// )
// )</span></code></pre>
<p>And an instance of a <code>TypedDataset</code>:</p>
Expand All @@ -167,7 +167,7 @@ <h2 id="example" class="section"><a class="anchor-link left" href="#example"><i
</span><span class="identifier">cal</span><span>
}
}
</span><span class="comment">// calendarToLongInjection: AnyRef with Injection[Calendar, Long] = repl.MdocSession$MdocApp0$$anon$1@10f4a4d6</span></code></pre>
</span><span class="comment">// calendarToLongInjection: AnyRef with Injection[Calendar, Long] = repl.MdocSession$MdocApp0$$anon$1@4ba9ca16</span></code></pre>
<p>We can be less verbose using the <code>Injection.apply</code> function:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">import</span><span> </span><span class="identifier">frameless</span><span>.</span><span class="identifier">_</span><span>

Expand All @@ -180,7 +180,7 @@ <h2 id="example" class="section"><a class="anchor-link left" href="#example"><i
</span><span class="identifier">cal</span><span>.</span><span class="identifier">setTime</span><span>(</span><span class="keyword">new</span><span> </span><span class="identifier">java</span><span>.</span><span class="identifier">util</span><span>.</span><span class="type-name">Date</span><span>(</span><span class="identifier">l</span><span>))
</span><span class="identifier">cal</span><span>
})
</span><span class="comment">// calendarToLongInjection: Injection[Calendar, Long] = frameless.Injection$$anon$1@5ad19282</span></code></pre>
</span><span class="comment">// calendarToLongInjection: Injection[Calendar, Long] = frameless.Injection$$anon$1@4bc98c72</span></code></pre>
<p>Now we can create our <code>TypedDataset</code>:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">val</span><span> </span><span class="identifier">personDS</span><span> = </span><span class="type-name">TypedDataset</span><span>.</span><span class="identifier">create</span><span>(</span><span class="identifier">people</span><span>)
</span><span class="comment">// personDS: TypedDataset[Person] = [age: int, birthday: bigint]</span></code></pre>
Expand Down Expand Up @@ -214,7 +214,7 @@ <h2 id="another-example" class="section"><a class="anchor-link left" href="#anot
</span><span class="keyword">case</span><span> </span><span class="number-literal">2</span><span> =&gt; </span><span class="type-name">Female</span><span>
</span><span class="keyword">case</span><span> </span><span class="number-literal">3</span><span> =&gt; </span><span class="type-name">Other</span><span>
})
</span><span class="comment">// genderToInt: Injection[Gender, Int] = frameless.Injection$$anon$1@583e4aa</span></code></pre>
</span><span class="comment">// genderToInt: Injection[Gender, Int] = frameless.Injection$$anon$1@40b834de</span></code></pre>
<p>And now we can create our <code>TypedDataset</code>:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">val</span><span> </span><span class="identifier">personDS</span><span> = </span><span class="type-name">TypedDataset</span><span>.</span><span class="identifier">create</span><span>(</span><span class="identifier">people</span><span>)
</span><span class="comment">// personDS: TypedDataset[Person] = [age: int, gender: int]</span></code></pre>
Expand Down
4 changes: 2 additions & 2 deletions Job.html
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ <h1 id="job-a" class="title">Job[A]</h1>
</span><span class="identifier">count</span><span> &lt;- </span><span class="identifier">ds</span><span>.</span><span class="identifier">count</span><span>()
</span><span class="identifier">sample</span><span> &lt;- </span><span class="identifier">ds</span><span>.</span><span class="identifier">take</span><span>((</span><span class="identifier">count</span><span>/</span><span class="number-literal">5</span><span>).</span><span class="identifier">toInt</span><span>)
} </span><span class="keyword">yield</span><span> </span><span class="identifier">sample</span><span>
</span><span class="comment">// countAndTakeJob: frameless.Job[Seq[Int]] = frameless.Job$$anon$3@5a507fe4
</span><span class="comment">// countAndTakeJob: frameless.Job[Seq[Int]] = frameless.Job$$anon$3@1bbad6ea
</span><span>
</span><span class="identifier">countAndTakeJob</span><span>.</span><span class="identifier">run</span><span>()
</span><span class="comment">// res1: Seq[Int] = WrappedArray(1, 2, 3, 4)</span></code></pre>
Expand All @@ -167,7 +167,7 @@ <h1 id="job-a" class="title">Job[A]</h1>
</span><span class="keyword">def</span><span> </span><span class="declaration-name">computeMinOfSample</span><span>(</span><span class="identifier">sample</span><span>: </span><span class="type-name">Job</span><span>[</span><span class="type-name">Seq</span><span>[</span><span class="type-name">Int</span><span>]]): </span><span class="type-name">Job</span><span>[</span><span class="type-name">Int</span><span>] = </span><span class="identifier">sample</span><span>.</span><span class="identifier">map</span><span>(</span><span class="identifier">_</span><span>.</span><span class="identifier">min</span><span>)

</span><span class="keyword">val</span><span> </span><span class="identifier">finalJob</span><span> = </span><span class="identifier">computeMinOfSample</span><span>(</span><span class="identifier">countAndTakeJob</span><span>)
</span><span class="comment">// finalJob: Job[Int] = frameless.Job$$anon$2@6ecc4c82</span></code></pre>
</span><span class="comment">// finalJob: Job[Int] = frameless.Job$$anon$2@42a483cd</span></code></pre>
<p>Now we can execute this new job by specifying a <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext@setJobGroup(groupId:String,description:String,interruptOnCancel:Boolean):Unit">group-id</a> and a description.
This allows the programmer to see this information on the Spark UI and help track, say,
performance issues.</p>
Expand Down
2 changes: 1 addition & 1 deletion TypedDatasetVsSparkDataset.html
Original file line number Diff line number Diff line change
Expand Up @@ -159,9 +159,9 @@ <h1 id="comparing-typeddatasets-with-spark-s-datasets" class="title">Comparing T
</span><span class="comment">// +---+---+
// | i| j|
// +---+---+
// | 1| Q|
// | 10| W|
// |100| E|
// | 1| Q|
// +---+---+
//</span></code></pre>
<p>The value <code>ds</code> holds the content of the <code>initialDs</code> read from a parquet file.
Expand Down
2 changes: 1 addition & 1 deletion TypedEncoder.html
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ <h1 id="typed-encoders-in-frameless" class="title">Typed Encoders in Frameless</
</span><span class="comment">// ds: TypedDataset[Foo] = [i: int, b: struct&lt;d: double, s: string&gt;]
</span><span>
</span><span class="identifier">ds</span><span>.</span><span class="identifier">collect</span><span>()
</span><span class="comment">// res3: frameless.Job[Seq[Foo]] = frameless.Job$$anon$4@33820cc8</span></code></pre>
</span><span class="comment">// res3: frameless.Job[Seq[Foo]] = frameless.Job$$anon$4@572adb</span></code></pre>
<p>But any non-encodable in the case class hierarchy will be detected at compile time:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">case</span><span> </span><span class="keyword">class</span><span> </span><span class="type-name">BarDate</span><span>(</span><span class="identifier">d</span><span>: </span><span class="type-name">Double</span><span>, </span><span class="identifier">s</span><span>: </span><span class="type-name">String</span><span>, </span><span class="identifier">t</span><span>: </span><span class="identifier">java</span><span>.</span><span class="identifier">util</span><span>.</span><span class="type-name">Calendar</span><span>)
</span><span class="keyword">case</span><span> </span><span class="keyword">class</span><span> </span><span class="type-name">FooDate</span><span>(</span><span class="identifier">i</span><span>: </span><span class="type-name">Int</span><span>, </span><span class="identifier">b</span><span>: </span><span class="type-name">BarDate</span><span>)</span></code></pre>
Expand Down
Loading

0 comments on commit 8aa4151

Please sign in to comment.