Impala
Impalaistheopensource,nativeanalyticdatabaseforApacheHadoop.
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros
Public Member Functions | Private Attributes | List of all members
impala::QuerySchedule Class Reference

#include <query-schedule.h>

Collaboration diagram for impala::QuerySchedule:

Public Member Functions

 QuerySchedule (const TUniqueId &query_id, const TQueryExecRequest &request, const TQueryOptions &query_options, const std::string &effective_user, RuntimeProfile *summary_profile, RuntimeProfile::EventSequence *query_events)
 
Status ValidateReservation ()
 
const TUniqueId & query_id () const
 
const TQueryExecRequest & request () const
 
const TQueryOptions & query_options () const
 
const std::string & effective_user () const
 
const std::string & request_pool () const
 
void set_request_pool (const std::string &pool_name)
 
bool HasReservation () const
 
bool NeedsRelease () const
 
int64_t GetPerHostMemoryEstimate () const
 
int16_t GetPerHostVCores () const
 
int64_t GetClusterMemoryEstimate () const
 Total estimated memory for all nodes. set_num_hosts() must be set before calling. More...
 
void GetResourceHostport (const TNetworkAddress &src, TNetworkAddress *dst)
 
void AddScanRanges (int64_t delta)
 Helper methods used by scheduler to populate this QuerySchedule. More...
 
void set_num_backends (int64_t num_backends)
 
void set_num_hosts (int64_t num_hosts)
 
int64_t num_backends () const
 
int64_t num_hosts () const
 
int64_t num_scan_ranges () const
 
int32_t GetFragmentIdx (PlanNodeId id) const
 
std::vector< FragmentExecParams > * exec_params ()
 
const boost::unordered_set
< TNetworkAddress > & 		unique_hosts () const
 
TResourceBrokerReservationResponse * reservation ()
 
const
TResourceBrokerReservationRequest & 		reservation_request () const
 
bool is_admitted () const
 
void set_is_admitted (bool is_admitted)
 
RuntimeProfile * summary_profile ()
 
RuntimeProfile::EventSequence * query_events ()
 
void SetUniqueHosts (const boost::unordered_set< TNetworkAddress > &unique_hosts)
 
void PrepareReservationRequest (const std::string &pool, const std::string &user)
 

Private Attributes

const TUniqueId & query_id_
 
const TQueryExecRequest & request_
 
const TQueryOptions & query_options_
 
const std::string effective_user_
 
RuntimeProfile * summary_profile_
 
RuntimeProfile::EventSequence * query_events_
 
std::vector< int32_t > plan_node_to_fragment_idx_
 Maps from plan node id to its fragment index. Filled in c'tor. More...
 
std::vector< FragmentExecParams > fragment_exec_params_
 
boost::unordered_set
< TNetworkAddress > 		unique_hosts_
 The set of hosts that the query will run on excluding the coordinator. More...
 
int64_t num_backends_
 Number of backends executing plan fragments on behalf of this query. More...
 
int64_t num_hosts_
 
int64_t num_scan_ranges_
 Total number of scan ranges of this query. More...
 
std::string request_pool_
 Request pool to which the request was submitted for admission. More...
 
TResourceBrokerReservationRequest reservation_request_
 Reservation request to be submitted to Llama. Set in PrepareReservationRequest(). More...
 
TResourceBrokerReservationResponse reservation_
 Fulfilled reservation request. Populated by scheduler. More...
 
bool is_admitted_
 Indicates if the query has been admitted for execution. More...
 
boost::scoped_ptr
< ResourceResolver > 		resource_resolver_
 

Detailed Description

A QuerySchedule contains all necessary information for a query coordinator to generate fragment execution requests and start query execution. If resource management is enabled, then a schedule also contains the resource reservation request and the granted resource reservation. TODO: Consider moving QuerySchedule and all Schedulers into their own lib (and out of statestore). TODO: Move all global state (e.g. profiles) to QueryExecState (after it is decoupled from ImpalaServer)

Definition at line 67 of file query-schedule.h.

Constructor & Destructor Documentation

impala::QuerySchedule::QuerySchedule ( const TUniqueId &  query_id,
const TQueryExecRequest &  request,
const TQueryOptions &  query_options,
const std::string &  effective_user,
RuntimeProfile *  summary_profile,
RuntimeProfile::EventSequence *  query_events 
)

Definition at line 59 of file query-schedule.cc.

References fragment_exec_params_, and plan_node_to_fragment_idx_.

Member Function Documentation

void impala::QuerySchedule::AddScanRanges ( int64_t  delta)
inline

Helper methods used by scheduler to populate this QuerySchedule.

Definition at line 100 of file query-schedule.h.

References num_scan_ranges_.

Referenced by impala::SimpleScheduler::ComputeScanRangeAssignment().

const std::string& impala::QuerySchedule::effective_user ( ) const
inline

Definition at line 80 of file query-schedule.h.

References effective_user_.

Referenced by impala::SimpleScheduler::Schedule().

std::vector<FragmentExecParams>* impala::QuerySchedule::exec_params ( )
inline

Definition at line 110 of file query-schedule.h.

References fragment_exec_params_.

Referenced by impala::SimpleScheduler::ComputeFragmentExecParams(), impala::SimpleScheduler::ComputeFragmentHosts(), and impala::SimpleScheduler::ComputeScanRangeAssignment().

int64_t impala::QuerySchedule::GetClusterMemoryEstimate ( ) const

Total estimated memory for all nodes. set_num_hosts() must be set before calling.

Definition at line 86 of file query-schedule.cc.

References GetPerHostMemoryEstimate(), and num_hosts_.

Referenced by impala::AdmissionController::AdmitQuery(), impala::AdmissionController::CanAdmitRequest(), impala::AdmissionController::RejectRequest(), and impala::AdmissionController::ReleaseQuery().

int32_t impala::QuerySchedule::GetFragmentIdx ( PlanNodeId  id) const
inline

Definition at line 109 of file query-schedule.h.

References plan_node_to_fragment_idx_.

Referenced by impala::SimpleScheduler::ComputeScanRangeAssignment().

int64_t impala::QuerySchedule::GetPerHostMemoryEstimate ( ) const

Gets the estimated memory (bytes) and vcores per-node. Returns the user specified estimate (MEM_LIMIT query parameter) if provided or the estimate from planning if available, but is capped at the amount of physical memory to avoid problems if either estimate is unreasonably large.

Definition at line 93 of file query-schedule.cc.

References impala::ParseUtil::ParseMemSpec(), impala::MemInfo::physical_mem(), query_options_, and request_.

Referenced by GetClusterMemoryEstimate(), and PrepareReservationRequest().

int16_t impala::QuerySchedule::GetPerHostVCores ( ) const

Definition at line 137 of file query-schedule.cc.

References query_options_.

Referenced by PrepareReservationRequest().

void impala::QuerySchedule::GetResourceHostport ( const TNetworkAddress &  src,
TNetworkAddress *  dst 
)

Definition at line 151 of file query-schedule.cc.

References resource_resolver_.

bool impala::QuerySchedule::HasReservation ( ) const
inline

Definition at line 83 of file query-schedule.h.

References reservation_.

Referenced by ValidateReservation().

bool impala::QuerySchedule::is_admitted ( ) const
inline

Definition at line 118 of file query-schedule.h.

References is_admitted_.

Referenced by impala::AdmissionController::ReleaseQuery(), and set_is_admitted().

bool impala::QuerySchedule::NeedsRelease ( ) const
inline

Granted or timed out reservations need to be released. In both such cases, the reservation_'s reservation_id is set.

Definition at line 87 of file query-schedule.h.

References reservation_.

Referenced by impala::SimpleScheduler::Release().

int64_t impala::QuerySchedule::num_backends ( ) const
inline

Definition at line 106 of file query-schedule.h.

References num_backends_.

Referenced by set_num_backends().

int64_t impala::QuerySchedule::num_hosts ( ) const
inline

Definition at line 107 of file query-schedule.h.

References num_hosts_.

Referenced by set_num_hosts().

int64_t impala::QuerySchedule::num_scan_ranges ( ) const
inline

Definition at line 108 of file query-schedule.h.

References num_scan_ranges_.

void impala::QuerySchedule::PrepareReservationRequest ( const std::string &  pool,
const std::string &  user 
)

Populates reservation_request_ ready to submit a query to Llama for all initial resources required for this query.

Definition at line 164 of file query-schedule.cc.

References impala::DEFAULT_REQUEST_TIMEOUT_MS, GetPerHostMemoryEstimate(), GetPerHostVCores(), llama::GetShortName(), pool, query_options_, reservation_request_, resource_resolver_, and unique_hosts_.

Referenced by impala::SimpleScheduler::Schedule().

RuntimeProfile::EventSequence* impala::QuerySchedule::query_events ( )
inline

Definition at line 121 of file query-schedule.h.

References query_events_.

Referenced by impala::AdmissionController::AdmitQuery().

const TUniqueId& impala::QuerySchedule::query_id ( ) const
inline

Definition at line 77 of file query-schedule.h.

References query_id_.

Referenced by impala::AdmissionController::AdmitQuery(), impala::SimpleScheduler::ComputeFragmentExecParams(), impala::AdmissionController::DequeueLoop(), and impala::AdmissionController::ReleaseQuery().

const TQueryOptions& impala::QuerySchedule::query_options ( ) const
inline

Definition at line 79 of file query-schedule.h.

References query_options_.

Referenced by impala::SimpleScheduler::ComputeScanRangeAssignment(), and impala::SimpleScheduler::Schedule().

const TQueryExecRequest& impala::QuerySchedule::request ( ) const
inline

Definition at line 78 of file query-schedule.h.

References request_.

Referenced by impala::SimpleScheduler::Schedule().

const std::string& impala::QuerySchedule::request_pool ( ) const
inline

Definition at line 81 of file query-schedule.h.

References request_pool_.

Referenced by impala::AdmissionController::AdmitQuery(), and impala::AdmissionController::ReleaseQuery().

TResourceBrokerReservationResponse* impala::QuerySchedule::reservation ( )
inline

Definition at line 114 of file query-schedule.h.

References reservation_.

Referenced by impala::SimpleScheduler::Release(), and impala::SimpleScheduler::Schedule().

const TResourceBrokerReservationRequest& impala::QuerySchedule::reservation_request ( ) const
inline

Definition at line 115 of file query-schedule.h.

References reservation_request_.

Referenced by impala::SimpleScheduler::Schedule().

void impala::QuerySchedule::set_is_admitted ( bool  is_admitted)
inline

Definition at line 119 of file query-schedule.h.

References is_admitted(), and is_admitted_.

Referenced by impala::AdmissionController::AdmitQuery().

void impala::QuerySchedule::set_num_backends ( int64_t  num_backends)
inline

Definition at line 101 of file query-schedule.h.

References num_backends(), and num_backends_.

Referenced by impala::SimpleScheduler::ComputeFragmentExecParams().

void impala::QuerySchedule::set_num_hosts ( int64_t  num_hosts)
inline

Definition at line 102 of file query-schedule.h.

References num_hosts(), and num_hosts_.

Referenced by impala::SimpleScheduler::Schedule().

void impala::QuerySchedule::set_request_pool ( const std::string &  pool_name)
inline

Definition at line 82 of file query-schedule.h.

References request_pool_.

Referenced by impala::SimpleScheduler::Schedule().

void impala::QuerySchedule::SetUniqueHosts ( const boost::unordered_set< TNetworkAddress > &  unique_hosts)

Definition at line 159 of file query-schedule.cc.

References resource_resolver_, unique_hosts(), and unique_hosts_.

Referenced by impala::SimpleScheduler::ComputeFragmentHosts().

RuntimeProfile* impala::QuerySchedule::summary_profile ( )
inline

Definition at line 120 of file query-schedule.h.

References summary_profile_.

Referenced by impala::AdmissionController::AdmitQuery().

const boost::unordered_set<TNetworkAddress>& impala::QuerySchedule::unique_hosts ( ) const
inline

Definition at line 111 of file query-schedule.h.

References unique_hosts_.

Referenced by SetUniqueHosts().

Status impala::QuerySchedule::ValidateReservation ( )

Returns OK if reservation_ contains a matching resource for each of the hosts in fragment_exec_params_. Returns an error otherwise.

Definition at line 214 of file query-schedule.cc.

References fragment_exec_params_, impala::ResourceResolver::GetResourceHostport(), HasReservation(), impala::FragmentExecParams::hosts, impala::Status::OK, reservation_, and unique_hosts_.

Referenced by impala::SimpleScheduler::Schedule().

Member Data Documentation

const std::string impala::QuerySchedule::effective_user_
private

Definition at line 136 of file query-schedule.h.

Referenced by effective_user().

std::vector<FragmentExecParams> impala::QuerySchedule::fragment_exec_params_
private

vector is indexed by fragment index from TQueryExecRequest.fragments; populated by Scheduler::Schedule()

Definition at line 145 of file query-schedule.h.

Referenced by exec_params(), QuerySchedule(), and ValidateReservation().

bool impala::QuerySchedule::is_admitted_
private

Indicates if the query has been admitted for execution.

Definition at line 170 of file query-schedule.h.

Referenced by is_admitted(), and set_is_admitted().

int64_t impala::QuerySchedule::num_backends_
private

Number of backends executing plan fragments on behalf of this query.

Definition at line 151 of file query-schedule.h.

Referenced by num_backends(), and set_num_backends().

int64_t impala::QuerySchedule::num_hosts_
private

Total number of hosts. Used to compute the total cluster estimated memory in GetClusterMemoryEstimate().

Definition at line 155 of file query-schedule.h.

Referenced by GetClusterMemoryEstimate(), num_hosts(), and set_num_hosts().

int64_t impala::QuerySchedule::num_scan_ranges_
private

Total number of scan ranges of this query.

Definition at line 158 of file query-schedule.h.

Referenced by AddScanRanges(), and num_scan_ranges().

std::vector<int32_t> impala::QuerySchedule::plan_node_to_fragment_idx_
private

Maps from plan node id to its fragment index. Filled in c'tor.

Definition at line 141 of file query-schedule.h.

Referenced by GetFragmentIdx(), and QuerySchedule().

RuntimeProfile::EventSequence* impala::QuerySchedule::query_events_
private

Definition at line 138 of file query-schedule.h.

Referenced by query_events().

const TUniqueId& impala::QuerySchedule::query_id_
private

These references are valid for the lifetime of this query schedule because they are all owned by the enclosing QueryExecState.

Definition at line 133 of file query-schedule.h.

Referenced by query_id().

const TQueryOptions& impala::QuerySchedule::query_options_
private

Definition at line 135 of file query-schedule.h.

Referenced by GetPerHostMemoryEstimate(), GetPerHostVCores(), PrepareReservationRequest(), and query_options().

const TQueryExecRequest& impala::QuerySchedule::request_
private

Definition at line 134 of file query-schedule.h.

Referenced by GetPerHostMemoryEstimate(), and request().

std::string impala::QuerySchedule::request_pool_
private

Request pool to which the request was submitted for admission.

Definition at line 161 of file query-schedule.h.

Referenced by request_pool(), and set_request_pool().

TResourceBrokerReservationResponse impala::QuerySchedule::reservation_
private

Fulfilled reservation request. Populated by scheduler.

Definition at line 167 of file query-schedule.h.

Referenced by HasReservation(), NeedsRelease(), reservation(), and ValidateReservation().

TResourceBrokerReservationRequest impala::QuerySchedule::reservation_request_
private

Reservation request to be submitted to Llama. Set in PrepareReservationRequest().

Definition at line 164 of file query-schedule.h.

Referenced by PrepareReservationRequest(), and reservation_request().

boost::scoped_ptr<ResourceResolver> impala::QuerySchedule::resource_resolver_
private

Resolves unique_hosts_ to node mgr addresses. Valid only after SetUniqueHosts() has been called.

Definition at line 174 of file query-schedule.h.

Referenced by GetResourceHostport(), PrepareReservationRequest(), and SetUniqueHosts().

RuntimeProfile* impala::QuerySchedule::summary_profile_
private

Definition at line 137 of file query-schedule.h.

Referenced by summary_profile().

boost::unordered_set<TNetworkAddress> impala::QuerySchedule::unique_hosts_
private

The set of hosts that the query will run on excluding the coordinator.

Definition at line 148 of file query-schedule.h.

Referenced by PrepareReservationRequest(), SetUniqueHosts(), unique_hosts(), and ValidateReservation().

The documentation for this class was generated from the following files: