Custom namespace for Job and related (Step, and maybe some *Reader/Writer) [BATCH-63]

[spring-projects-issues]

Dave Syer opened BATCH-63 and commented

StepScope needs to be validated - certain collaborators (like implementations of InputSource) have to be scope="step" and it is quite hard to validate. Lots of docos needed. A minimal custom namespace just for Tasklet configuration might be the most practical solution.

---

Affects: 1.0-m2

Attachments:

• Batch Namespace - Spring Batch - SpringSource Wiki.webarchive (278.68 kB)
• dsl-example-hibernate.xml (1.43 kB)
• dsl-example-trades.xml (2.37 kB)
• jobs.xml (1.64 kB)
• spring-batch-1.0.m4.xsd (9.82 kB)
• SpringBatch-6099862-20080218-16_00_28.pdf (175.81 kB)

Referenced from: commits 447b1f0, 190b14b, 3b0331a, c99fa10, 5274b55, ef00a5e, 0d4777e, 816c4b2, 13734f0, f8a62d6, d49cc44, 97ce100, a21f0a2, a5107fb, 8a1eae3, 24eab0a, 79c8465, ac6493b, 760b040, 315f08a, dc656d2, 37bfe92, dfa783a, c714494, feeb5bd

[spring-projects-issues]

Dave Syer commented

Technical hitch - inner beans cannot be scoped...

This is turning into a documentation issue - no-one has time / technology to fix it properly.

[spring-projects-issues]

Dave Syer commented

The technical hitch is a bug SPR-3800 which should be fixed in 2.1-m4 (N.B. inner beans can be scoped - they were just not registering their destruction callbacks). But this whole issue is still not looking like it is going to be fixed for 1.0. Documentation covered elsewhere (BATCH-120).

[spring-projects-issues]

Lucas Ward commented

Dave,

Can you attach the example namespace we came up with last time I was in SH, for others to be able to review? I seem to have lost it in all my computer changes.

[spring-projects-issues]

Lucas Ward commented

I did a quick review of the schema that Ben attached recently. Everything looks good except for the database ItemReader. I see a Sql reader(should probably be jdbc reader), with a type of cursor or driving, and an iBatis reader. My issue is that the iBatis reader is really a driving reader. More specifically, it's a driving query reader with an iBatis generator. Normally, it wouldn't be too bad, but I think the difference between the two types is big enough to warrant a more explicit separation. This is because the cursor item reader returns the whole item, whereas it's generally expected that the driving one will only return the key, requiring more information to be loaded via another dao.

Everything else looks fine, although I'm curious how you plan to handle the column mappings in file input, since I don't see them in the prototype.

Lastly, one use case that needs to be maintained is the ability to set default settings for batch jobs that may be overridden. We currently do this by using Abstract beans with the default values, and child beans that can override them. I'm not sure how to best handle this with a namespace, but in my mind it's an extremely important use case to keep. If someone has even as few as 10 batch jobs, you wouldn't want to wire in exception handler (for logging out skipped records) separately for each job. it becomes even worse in the very common case where 50+ distinct batch jobs exist.

[spring-projects-issues]

nebhale commented

(In reply to comment #4)

I did a quick review of the schema that Ben attached recently. Everything looks
good except for the database ItemReader. I see a Sql reader(should probably be
jdbc reader)

We opted for SQL since from the perspective of the user, they don't actually deal with JDBC. This actually was discussed a bit and Dave and I both agree that SQL is more appropriate in this case.

, with a type of cursor or driving, and an iBatis reader. My issue
is that the iBatis reader is really a driving reader. More specifically, it's a
driving query reader with an iBatis generator. Normally, it wouldn't be too
bad, but I think the difference between the two types is big enough to warrant a
more explicit separation. This is because the cursor item reader returns the
whole item, whereas it's generally expected that the driving one will only
return the key, requiring more information to be loaded via another dao.

What kind of separation do you have in mind between a driving reader and a cursor reader? Basically from a configuration perspective, both of them require a query and then we take a look at the flag to determine which type of ItemReader to actually instantiate. The reason that iBatis is kept separately is because it doesn't actually require a query at all, it requires an iBatis named query. If you think this is still too confusing I'd be game for removing the <ibatis-item-reader/> tag completely as I think it's a pretty small use case. In the end the data being loaded by a separate DAO is actually just an implementation detail of your transformer rather than of the reader, so I'm pretty happy with the way this is configured now.

Everything else looks fine, although I'm curious how you plan to handle the
column mappings in file input, since I don't see them in the prototype.

You're absolutely right on this. I do plan on letting this shake out as I actually do the implementations of those projects.

Lastly, one use case that needs to be maintained is the ability to set default
settings for batch jobs that may be overridden. We currently do this by using
Abstract beans with the default values, and child beans that can override them.
I'm not sure how to best handle this with a namespace, but in my mind it's an
extremely important use case to keep. If someone has even as few as 10 batch
jobs, you wouldn't want to wire in exception handler (for logging out skipped
records) separately for each job. it becomes even worse in the very common case
where 50+ distinct batch jobs exist.

Well, I think that this is more of an issue in cases where we use very verbose configurations (like bean definitions). I think that simply having a ref to a bean name in a number of places is fine in the case. At the very least, I'd like to go to 1.0 without explicit support for abstract namespace elements and if in practice people dislike configuring each time (again, just a ref to a separate bean, not a full bean definition) then we can add it in. I'm very concerned about the backwards compatibility of the namespace, so I want to go with a minimal set of support in this case.

Dave, do you have any comments on any of this?

[spring-projects-issues]

nebhale commented

Working version of Spring Batch namespace. You'll need either Mac or Windows Safari to open it.

[spring-projects-issues]

Scott Wintermute commented

Ben, do you mind exporting the Spring Batch namespace as pdf for those of us who aren't using a Mac or Safari. I would prefer not to load another browser and Confluence should let you do an easy export of the page in PDF. Let me know if this is an issue.

[spring-projects-issues]

nebhale commented

The reason that I chose the other way is because the confluence export isn't very good. I've attached it though.

[spring-projects-issues]

Lucas Ward commented

We opted for SQL since from the perspective of the user, they don't actually deal with JDBC. This actually was discussed a bit and Dave and I both agree that SQL is more >appropriate in this case.

I disagree with this, since it's not used commonly in spring. For example, if you use JdbcTemplate, you don't have to work with Jdbc directly either, and certainly no more than you do with the JdbcCursorItemReader.

I also disagree with the comments about Driving vs. Cursor, there is a huge difference between the usage: Cursor returns an item, and driving returns a key (expecting you to create an item from it) It's a HUGE difference in usage. Furthermore, the difference between the two types should be understood by anyone wanting to read in from a database, as they could have a lot of impacts depending upon which database type you're using. I would much rather have two separate tags for driving and cursor than one with a 'switch' for the two. Another option may be just leaving ItemReaders out of the namespace for release 1.0.

Regarding 'abstract configuration. I have very strong feelings about this one, since we already have a lot of projects, some in production, that would be unable to use the namespace if support for this is not added. It is an extremely important use-case to allow projects to do things like set a default strategy for logging out skipped items (take a look at the chunkedStep for more details on that)

[spring-projects-issues]

Douglas C. Kaminsky commented

I don't see the value in including the infrastructure components (i.e. ItemReaders and ItemWriters) as schema elements at all as opposed to just using beans of the required type (enforced via the tools namespace). Here's my thinking:

1. The infrastructure components may change in code much faster than the schema can change. For example, if we release version 1.0.1 of the framework with an added flag for some item reader, we can't keep changing the published schema, thus more or less implicitly deprecating the tag.

2. More infrastructure components are bound to be added after the 1.0 release with no schema support. How do we decide what's important enough to create a tag for and what is not? It seems arbitrary.

3. We have to include support for a beans:bean anyway to allow insertion of custom item readers / writers. Type safety is already addressable via the tools schema.

4. With the amount of configurable data for a given item reader or writer, a custom tag doesn't buy us much clarity - e.g. key generators for driving query readers.

5. We can always add it for a 1.5 or 2.0 release with an accompanying schema.

[spring-projects-issues]

Douglas C. Kaminsky commented

Just realized I posted before commenting on the abstraction issue - using abstract beans is EXTRAORDINARILY important for managing configuration. I just came off of a project where the projected number of batch jobs numbers in the thousands. Without the ability to abstract common aspects of the configurations, Spring Batch becomes exponentially less useful.

[spring-projects-issues]

nebhale commented

(In reply to comment #9)

I disagree with this, since it's not used commonly in spring. For example, if
you use JdbcTemplate, you don't have to work with Jdbc directly either, and
certainly no more than you do with the JdbcCursorItemReader.

This was actually a recommendation from Juergen himself. He said that from his perspective, it was an SQL only situation and he'd change it in Spring if he could. The first rev of the schema actually had data-source-reader there and when presented with jdbc-reader or sql-reader, he chose the latter.

I also disagree with the comments about Driving vs. Cursor, there is a huge
difference between the usage: Cursor returns an item, and driving returns a key
(expecting you to create an item from it) It's a HUGE difference in usage.

But this is an implementation detail. From the configuration perspective (and the left side of the chunk) the only thing that is required is a query and whether it is driving or not. When it comes to what is returned from the ItemReader, the code doesn't really care until after it's passed to the transformer. Since the transformer is written by the user, that detail is easily dealt with there. The configuration never has to care.

Another option may be just leaving ItemReaders out of the namespace for release 1.0.

This was considered for the 1.0 release, but as most of the configuration would then lie in the readers, we opted to have some sort of simplification.

Regarding 'abstract configuration. I have very strong feelings about this one,
since we already have a lot of projects, some in production, that would be
unable to use the namespace if support for this is not added.

The problem is of course that the concept of abstract namespace elements isn't really supported. The concept of the abstract bean is supported at the container level but a namespace element doesn't map one-to-one with a bean, so it doesn't really work out.

It is an extremely important use-case to allow projects to do things like set a default
strategy for logging out skipped items (take a look at the chunkedStep for more
details on that)

This I think gets more to the crux of the argument. What about the addition of a <default-step-configuration> element that talks about default strategies rather than the concept of an abstract bean?

[spring-projects-issues]

Lucas Ward commented

Ben,

"This I think gets more to the crux of the argument. What about the addition of a <default-step-configuration> element that talks about default strategies rather than the concept of an abstract bean?"

That is completely fine, I didn't mean that it had to support abstract beans, just that the ability to create default settings is an important use case. Being able to create a 'default step' is much much better actually.

@Sql vs. Jdbc
I don't mind too much honestly, but if we're going to call the tag Sql* instead of Jdbc*, I would prefer to change the readers back to Sql* as well.

"But this is an implementation detail. From the configuration perspective (and the left side of the chunk) the only thing that is required is a query and whether it is driving or not. When it comes to what is returned from the ItemReader, the code doesn't really care until after it's passed to the transformer. Since the transformer is written by the user, that detail is easily dealt with there. The configuration never has to care."

I understand your point about the 'Writers' or 'Readers' dealing with the output of the readers as a separate concern from the configuration. However, my point is that there is a huge domain difference between the 'driving' and 'cursor' based readers. It's so important in fact, that I think it should be separated. I'm extremely concerned that someone will try and use a type of 'driving' but then use the same type of sql statement and RowMapper that they would use for cursor (i.e. for returning the item rather than a key) This would lead to every issue that Rob highlighted so well at TSE, since the DrivingQueryItemReader holds the keys in a list. I understand your point about trying to make them less verbose, but I'm not sure that trying to combine the different types in configuration is a good idea. I would instead prefer to keep them separated, as they are in the code by driving vs. cursor. and I would definitely like to have a '<driving-query-item-reader> that has a <key generator> tag. I know that documentation can help with some of these concerns, but I still feel like it's much better to be explicit.

[spring-projects-issues]

nebhale commented

(In reply to comment #10)

I don't see the value in including the infrastructure components (i.e.
ItemReaders and ItemWriters) as schema elements at all as opposed to just using
beans of the required type (enforced via the tools namespace). Here's my
thinking:

This had actually been considered.

1. The infrastructure components may change in code much faster than the schema
   can change. For example, if we release version 1.0.1 of the framework with an
   added flag for some item reader, we can't keep changing the published schema,
   thus more or less implicitly deprecating the tag.

Well, we split the item-reader/writers off into another schema just so it could change quickly. It has been pointed out that a good factory could hide a lot of the complexity of the implementations thus negating the need for a namespace at all.

2. More infrastructure components are bound to be added after the 1.0 release
   with no schema support. How do we decide what's important enough to create a tag
   for and what is not? It seems arbitrary.

Well, I'd say it's driven by community demand. In one of the earlier comments I mentioned that I would not be averse to removing the ibatis support as it is a pretty small minority of users.

3. We have to include support for a beans:bean anyway to allow insertion of
   custom item readers / writers. Type safety is already addressable via the tools
   schema.

We actually don't. We discussed having the ability to define an in-line bean, but to stay with the pattern set out by Spring, we were only going to allow references to other beans. In fact, Juergen pointed out that pretty much the only namespace that allows in-line bean definitions is 'beans'.

4. With the amount of configurable data for a given item reader or writer, a
   custom tag doesn't buy us much clarity - e.g. key generators for driving query
   readers.

Fair point. The issue is that with a namespace we can more clearly enumerate (and in some cases create a more concise definition for) the dependencies. That being said, I'm not against this.

5. We can always add it for a 1.5 or 2.0 release with an accompanying schema.

And that's why :)

[spring-projects-issues]

nebhale commented

(In reply to comment #11)

See Comment #12